General Disclaimer 


One or more of the Following Statements may affect this Document 


• This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 


• This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 


• This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 


• This document is paginated as submitted by the original source. 


• Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 


Produced by the NASA Center for Aerospace Information (CASI) 



<^UXJJ04398 
JSC- 18602 


Ag R (STARS 


E83‘10406 


•‘Made available under NASA sponsorship 

'.i, ttie interest ot early ami wiJv dis- 
s, njn.itioii ol Edith Resources Survey 
p-o^ram intormation amt wulioul liabilitl 

lrivl§i1l(3il^"Yechnology 

Development 


tH-h! 67S" 

A Joint Program for 
Agriculture and 
Resources Inventory 
Surveys Through 
Aerospace 
Remote Sensing 


April 1983 


THE 1980 U.S./CANADA WHEAT AND BARLEY EXPLORATORY 
EXPERIMENT FINAL REPORT - ADDENDA 


VOLUME II 


/?. M. Bizzell and H. L Prior 


National Aeronautics and Space Administration 

(£83- 10406) THE 1980 OS/CANADA WHEAT AND N83-34399 

BARLEY EXPLORATORY EXPERIMENT. VOLDME 2: 

ADDENDA Fiaal Report (Lockheed Engineering 

and Manageiitent) 158 p HC A08/MF AO 1 Onclas 

CSCL 02C G3/43 00406 

'te? ** cpf 

/?. W, Payne and 1 M. Dialer 

'^Loctiheed Englneerlns and Management 1 1 

Sarytces Company, //w. 




o» 


Earth Rasourcsi Applications Division 
Lyndon 8. Johnson Space Canter 
Houston, Texas 77058 




ORlfiirAl. PA V r< 

OF POOK QJALlW 


1. Rtport No. 2. Gci>omwnt ReoHon No. 

JSC- 18602; IT-L3-04398 

3. Rddpwnt'i Cmtog No. 

4. Tttlt ond Subtitit 

The 1980 U.S. /Canada Wheat and Barley Exploratory Experiment 
Final Report, Volumes I and II 

6. Riport D»tt 
April 1983 

6. NHorming OftMnitation Codt 

7. AuttoorU) 

R. W. Payne and J. M. Disler, Lockheed 
R. M. Bizzell and H. L. Prior, NASA-OSC 

8. Nrtaiming Orgmiotien Rwen No. 
LEMSCO- 18629 

10. Work Unit No. 

9. ta^ormins OrgHiiatton Nmw and Addraa 

Lockheed Engineering and Management Services Co., Inc. 
1830 NASA Road 1 
Houston, Texas 77258 

11. Comnet or Gnm No. 
NAS 9-15800 

13. Typi of Roport mid Rmiod Covmod 
Final Report 

12. SpofMoring AgMcy Naim and Addian 

National Aeronautics and Space Administration 
Lyndon B. Johnson Space Center 

Houston, Texas 77058 Technical Monitor; J. L. Dragg 

14. Spomoring Agmiey Cod* 

IS. Suppiafnantarv New 

Tht Agrieulture and Rtsourees Inventory Surveys Through Aerospace Rwote Sensing Is a Joint program 
of the U.S. Department of Agriculture, the National Aeronautics and Space Administration, the National 
Oceanic and Atmospheric Administration (U.S. Department of Comaerce). the Agency for Inteniatlonal 
Develooment (U.S. Department of State), and the U.S. Department of the Interior. 


16. AMaei 


This report presents the results from the U.S, /Canada Wheat and Barley Exploratory 
Experiment which was completed during FY 1980. The results indicate that the new crop 
identification procedures performed well for spring small grains and that they are 
conducive to automation. The performance of the machine processing techniques shows a 
significant improvement over previously evaluated technology. However,, the crop calendars 
will require additional development and refinements prior to integration into automated 
area estimation technology. 

The evaluation has shown the integrated technology to be capable of producing accurate and 
consistent spring small grains proportion estimates. However, barley proportion estimation 
technology was not satisfactorily evaluated. Landsat sample segment data were not 
available for high-density ba-ley of primary importance in foreign regions. The 
low-density segments examined were judged not to give indicative or unequivocal results. 

It is concluded that, generally, the spring small grains technology is ready for evaluation 
in a pilot experiment focusing on sensitivity analyses to a variety of agricultural and 
meteorological conditions representative of the global environment. It is further 
concluded that a strong potential exists for establishing a highly efficient technology for 
spring small grains. 


Crop devel op^ot 
Analyst labeling stage models 

Barley Planting date models 

Bayesian estimator Remote sensing 

CLASSY stratification Satellite 

Crop Calendars Agri culture/Wheat 

18. Dinritoiftion Stattmant 

19. Sieiirity Oamif. (of thtonpert) 

Unclassified 

20. Saeurity Oamif. <of ttoia paga) 

Unclassified 

21, No. of hM* 

Vo1 . 1 - W 
Vol. II - 153 

22. Wea* 


*For Mil toy 6w NMienal Tactonieil InfennMan SttviM, SpringfWd, Vtoviiito 22161 


JIC Mrm 14M IRwMavTW 














IT-L3-04398 

JSC-18602 


ORM-iK'At. '"•/ ' ■ 

OF POOR ~rr/ 


THE 1980 U.S. /CANADA WHEAT AND BARLEY EXPLORATORY EXPERIMENT 

FINAL REPORT VOLUME II 


Job Order 72-418 


This report describes the 1980 U.S. /Canada Wheat and Barley Exploratory 
Experiment of the Inventory Technology Development Department within the 
AgRISTARS program. 


PREPARED BY 

R. M. Bizzell and H. L. Prior, NASA-JSC 

and 

R. W. Payne and J. M. Disler, Lockheed-EMSCO 


APPROVED BY 


NASA 


X ^'f\r^k'^Hroj«f’flanager 

Inventory Technology Development 


LOCKHEED-EMSCO 


1. D. Pore, Manager 

Inventory Technology Development 

Department 


LOCKHEED ENGINEERING AND MANAGEMENT SERVICES COMPANY, INC. 
Under Contract NAS 9-15800 
For 

Earth Resources Applications Division 

NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 
LYNDON B. JOHNSON SPACE CENTER 
HOUSTON, TEXAS 


April 1983 


LEMSCO-18629 


PREFACE 


The Agriculture and Resources Inventory Surveys Through Aerospace Remote 
Sensing is a program of research, development, evaluation, and application of 
aerospace remote sensing for agricultural resources. This program is a coop- 
erative effort of the National Aeronautics and Space Administration, the U.S. 
Departments of Agriculture, Commerce, and the Interior, and the U.S. Agency 
for International Development, 

The work which is the subject of this document was performed by the Earth 
Resources Applications Division, Space and Life Sciences Directorate, Lyndon 
6. Johnson Space Center, National Aeronautics and Space Administration and the 
Lockheed Engineering and Management Services Company, Inc. Tasks performed by 
Lockheed were accomplished under Contract NAS 9-15800. 


preceding page blank not filmed 


CONTENTS, VOLUME II 


Addendum 

1. EVALUATION OF THE U.S. /CANADA WHEAT AND BARLEY EXPLORATORY EXPERIMENT 
SHAKEDOWN TEST ANALYST LABELING RESULTS. JSC-17815, LEMSCO-16633, 
DEC. 1981. 

2. EVALUATION OF THE PROCEDURE lA COMPONENT OF THE 1980 U.S. /CANADA 
WHEAT AND BARLEY EXPLORATORY EXPERIMENT. JSC-17806, LEMSCO-16311, 
DEC. 1981. 

3. EVALUATION OF SPRING WHEAT AND BARLEY CROP CALENDAR MODELS FOR THE 
1979 CROP YEAR. JSC-16850, LEMSCO-15936, FEB. 1981. 


PRECEDING PAGE BLANK NOT FILMED 




vii 


ADDENDUM 1 


AgRISTARS 

< 


Foreign Commodity 
Production Forecasting 


FC-L2-04229 

JSC-17815 


A Joint Program for 
Agriculture and 
Resources Inventory 
Surveys Through 
Aerospace 
Remote Sensing 

December 1981 


EVALUATION OF THE U.S./C/^DA WHEAT AND BARLEY 
EX PLORATORY EX PERI MEN T SHAKEDOWN TEST 
ANALYST LABELING RESULTS 


J. G. Carnes 




Lockheed Engineering and Management Services Company, Inc. 
1830 NASA Road 1, Houston, Texas 77258 



Lyndon B. Johnson Space Center 

Houston, Texas 77058 


ORIGWAL FA3^: JS 
OF POOR QUALITY 


1. Naeart No. 2. fioMomfiwnt Acctnion No. 

FC-L2-04229! JSC-17815 

3. haetownCi Catalog No. 

4 Titit and SuMint 

Evaluation of the U.S. /Canada Wheat and Barley Exploratory 
Experiment Shakedown Test Analyst Labeling Results 

8. Xapon Data 
December 1981 

6. Atflorniing Organisation CoOt 

7. AythorUI 

J. G. Carnes 

a. Parterming Orgamtation haaon No 
LEMSCO- 16633 

10. Work Unit No. 

9. Nrfcrmine Organiation Nama and Addraai 

Lockheed Engineering and Management Services Company, Inc. 
1830 NASA Road 1 
Houston, Texas 77058 

11. Contract or Cram No. 

NAS 9-15800 

13. Typa of hapon and Pariod Covarad 
Technical Report 

12. Seonaortng Aganey Nama «id Addrau 

National Aeronautics and Space Administration 
Lyndon 8. Johnson Space Center 

Houston, Texas 77058 Technical Monitor: J. L. Oragg 

14. Soenaonng Aganev Coda 

IS. Susotamamarv Notat 


The Agriculture and Resources Inventory Surveys Through Aerospace Remote Sensing 1$ a Joint program 
of the U.S. Department of Agriculture, the National Aeronautics and Space Administration, the National 
Oceanic and Atmospheric Administration (U.S. Department of Commerce), the Agency for International 
. Development (l.S. Deoartment of State), and tht U.S. Department of the Interior. 

IS. ABnaei 

The results of the 1980 U.S. /Canada Wheat and Barley Exploratory labeling Experiment 
Shakedown Test are presented In this document. This test was performed on six sepents 
from the 1978 crop year. No major problems were found with the Reformatted Labeling 
Procedure as it was applied to these segments.. The labeling accuracies obtained using this 
procedure were comparable with the labeling accuracies obtained using the Integrated 
Labeling Procedure, ^hls Indicates that the procedure Is ready for testing with the 
recommended * changes . 


17. ttey Werda (Stiggaatad by Autnertatl 

Exploratory experiment 
Labeling accuracy evaluation 
Reformatted labeling procedure 
Small grains labeling procedures 
1980 U.S. /Canada Wheat and Barley 

ia ONiribution Statamam 


ie. Sacurity CUaaif. (of thia raeoril 
Unclassified 

20. 

Sacurity Oaaaif. (of tbit page) 

Unclassified 

21. No. of Pagat 
72 

22. Pnea* 


Toram bv ew Nnraiwl Tatmieil Infermatien Suviet, SprtngfiMd, Virgini* 8101 












FC-L2-04229 

JSC-17815 


EVALUATION OF THE U.S. /CANADA WHEAT AND BARLEY 
EXPLORATORY EXPERIMENT SHAKEDOWN TEST 
ANALYST LABELING RESULTS 

Job Order 72-422 

This report describes performance evaluation activities of the 
Foreign Commodity Production Forecasting of the AgRISTARS program, 

PREPARED BY 
J. G. Carnes 



L. Carroll, Manager 
Crop Applications Department 


LOCKHEED ENGINEERING AND MANAGEMENT SERVICES COMPANY, INC. 
Under Contract NAS 9-15800 
For 

Earth Resources Applications Division 

Space and Life Sciences Directorate 

NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 
LYNDON B. JOHNSON SPACE CENTER 
HOUSTON, TEXAS 

December 1981 


LEMSCO-16633 


PREFACE 


The Agriculture and Resources Inventory Surveys Through Aerospace Remote 
Sensing is a multiyear program of research, development, evaluation, and appli- 
cation of aerospace remote sensing for agricultural resources, which began in 
fiscal year 1980. This program is a cooperative effort of the U.S. Department 
of Agriculture, the National Aeronfiulics and Space Administration, the National 
Oceanic and Atmospheric Administration (U.S. Department of Commerce), the 
Agency for International Development (U.S. Department of State), and the 
U.S. Department of the Interior. 

The work which is the subject of this document was performed by the Earth 
Resources Applications Division, Space and Life Sciences Directorate, Lyndon B. 
Johnson Space Center, National Aeronautics and Space Administration and 
Lockheed Engineering and Management Services Company, Inc. The tasks performed 
by Lockheed Engineering and Management Services Company, Inc., were 
accomplished under Contract NAS 9-15800. 
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1. INTRODUCTION 


The objective of the Foreign Commodity Production Forecasting (FCPF) project 
of the Agriculture and Resources Inventory Surveys Through Aerospace Remote 
Sensing (AgRISTARS) program is to develop and test procedures for using aero- 
space and related technology. Specifically, this testing and development is 
done to provide more objective and reliable crop production forecasts several 
times during the growing season and to provide improved preharvest estimates 
for a range of countries and crops. During the first year of the project 
(1980), an exploratory study of at-harvest crop proportion estimates from 1979 
Landsat data for spring small grains was conducted on 5- by 6-naut1 cal -mile 
segments within the northern Great Plains. To produce segment-level estimates 
for this expei iment, analysts identify and label target pixels (dots) which 
are taken from Landsat Imagery. Usually these dots are taken from the set of 
209 pixels at the Intersection of every tenth line and every tenth sample in a 
line. 

In one procedure vor labeling these dots, the analyst assimilates information 
from image products, spectral aids, crop calendars, and assorted meteorol ogi- 
val and agronomic data. The analyst then subjectively applies weights to 
chese data to arrive at a label for the dot. This method of labeling dots is 
part of the Integrated Labeling Procedure (ref. 1) which was used during the 
Large Area Crop Inventory Experiment (LACIE) and the Transition Year. The 
accuracy of labeling using this method depends to a great extent on the 
ingenuity of the analyst doing the labeling. The results can vary greatly 
from analyst to analyst. However, because of the subjective nature of the 
technique and the amount of information examined, maximum use can be made of 
the available data. One problem with a subjective procedure is that it is not 
always obvious how the label is obtained. If the label is incorrect, one can 
only speculate as to the reason for the error. 
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Because of these undesirable features, it was recognized that a more system- 
atic and objective labeling procedure was? required. If a systematic labeling 
procedure could be developed, the skill requirements for analysts could be 
reduced, the resulting labels would be less variable, and the reasons for 
errors would be more easily identified. In addition, the analyst activities 
required to produce proportion estimates for sample segments could be 
significantly reduced or eliminated by automation. 

The Reformatted Labeling Procedure (see appendix A), for wheat and barley was 
developed to meet these requirements. It is based on a decision tree labeling 
logic. The labeling decision is obtained by answering a series of questions, 
with the answer to one question leading to the next question, until the end of 
the decision path is reached. The end point of the path determines the final 
label. By recoruing the answers to each question involved in the decision 
logic, it is possible to determine not only whether the label is correct but 
why incorrect labels were obtained. 

The U.S. /Canada Wheat and Barley Exploratory Labeling Experiment (ref. 2) 
provides the first evaluation of the labeling logic in the Reformatted 
Labeling Procedure. In this experiment, both the Reformatted and Integrated 
Labeling Procedures were used to produce dot labels using Landsat data from 
two crop years (1978 and 1979). 

There were two tests perfO"med in this labeling experiment. The first test 
(Shakedown Test) was performed using a limited number of segments from the 
1978 crop year. The Reformatted Labeling Procedure was developed using data 
from this crop year. However, the six segments involved in the Shakedown Test 
were not used in developing the procedure. The main purpose of the test was 
to cetermine if there v.ere any major problems with the procedure before it was 
applied in the second test to segments from the 1979 crop year. This was to 
be determined from the labeling accuracy and not from proportion estimation 
accuracy. The study of proportion estimation was the subject of a 
supplemental experiment. 
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The second test (Test 2) was designed to be a more extensive test of the 
procedure using data from a different crop year (1979). In this test the 
Integrated Labeling Procedure was applied to the same segments as the 
Reformatted Labeling Procedure. This provided a standard for comparison. 

This report, however, presents a brief description of the procedure and the 
results of the first test only. 
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2. DESCRIPTION OF THE REFORMATTED PROCEDURE 


To understand the results of this evaluation, one must have knowledge of the 
steps in the procedure. A detailed explanation of these steps is given in 
reference 2. This section will provide an outline of the procedure. 

The Reformatted Labeling Procedure is based on a decision tree labeling logic 
which produces progressively more detailed dot labeling. The first step in the 
procedure is to determine which Landsat acquisitions should be used in the 
decision process. On the basis of crop calendar information, four acquisition 
windows are defined. If an acquisition is available in one of these windows, 
it is used in the decision process. The following list indicates the biostages 
corresponding to the four acquisition windows (biostage lengths are shown in 
parentheses). 

1. pre-emergence for spring wheat (23 days) 

2. spring wheat, headed (20 days) 

3. barley, ripe (12 days) 

4. spring wheat, harvested (15 days) 

An additional window (time period A) is defined between windows 3 and 4. 

The major steps in the decision logic are shown in figure 2-1. The first 
decision separates the spectrally pure dots from the impure dots. The impure 
dots are those which exhibit more than one crop signature for the acquisitions 
used. Dots may be impure because they are on the borders of fields or because 
the acquisitions are not adequately registered with each other. Only the pure 
dots are labeled by the procedure^. 

^For labeling impure dots, a different approach was used. First, the pure dots 
were labeled. Second, through examination of the Landsat imagery, the analyst 
determined the field with which to associate the impure dot based on the 
spatial and spectral characteristics of the impure dot and adjacent fields. 

A comparison was then made between the multitemporal spectral signatures of 
the field associated with the impure dot and the fields within which pure dots 
had been labeled earlier. A labeling decision on the impure dot was then made 
on the basis cf the closest subjective matching. 
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Figure 2-1.- Shakedown Test results for the reformatted procedure. 
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The second major step in the decision logic separates the pure dots into tliose 
with cropland signatures and those with noncropland signatures. The logic 
involved in this step is based on the color of the dot on the production film 
converter (PFC) product [figure 2-2, (ref. 3)]. The path used to arrive at 
the cropland/noncropland decision is defined by the answers to questions la, 
lb, Ic, 2, 3, and 4 (see figure 2-1). The noncropland dots are labeled as 
nonsmall grains. 

The third major step separates the dots labeled cropland into those with small 
grains signatures and those with nonsmall- grains signatures. The logic in 
this step involves the green number (refs. 4 and 5) and brightness for the dot 
on each of the acquisitions (figure 2-3). Each of the green number and 
brightness decisions is given a number so that the path taken through the 
decision logic can be identified. 

The fourth major step separates the dots labeled small grains into those with 
barley signatures and those with signatures corresponding to other small 
grains (ref. 6). This decision is based on a green number versus brightness 
plot of the small grains dots for the acquisition in window 3. 
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Figure 2-2.- Decision logic for cropland and noncropland 
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Legend: 

DR • brightness 
GN > green nwiber 



Figure 2-3.- Decision logic for pure cropland dots 



3. SHAKEDOWN TEST WITH 1978 DATA 


3.1 EXPERIMENT DESIGN 

In the Shakedown Test, all 209 dots for six sejments were labeled using data 
from the 1978 crop year. The actual number of dots evaluated per segment 
varied downward from 209 because of clouds, cloud shadows, data dropouts, 
striping, or missing ground-truth inventory. The loss was a small percentage 
of the dots. The locations of the segments are shown in figure 3-1. Each of 
the segments was labeled by two analysts working independently. By comparing 
the two sets of labeling results, the consistency of the procedure could be 
evaluated. Five of these six segments were previously processed using the 
Integrated Labeling Procedure (ref. 7). These labeling results were used to 
compare the accuracy of the Reformatted Labeling Procedure with the accuracy 
of the Integrated Labeling Procedure. 

3.2 OVERALL LABELING ACCURACY FOR FINAL LABELS 

Table 3-1 shows the labeling accuracy for each of the categories labeled (non- 
small grains, barley, and other small grains). The labeling accuracy is shown 
for all the dots labeled and for those dots which were determined by the anal- 
yst to be pure, mixed, or misregistered. The labeling accuracy was greater 
for the pure dots (which were labeled using the decision logic) than for the 
impure dots (which were labeled by comparison with the pure dot labels). The 
numbers in parentheses show the percentage of dots correctly labeled when both 
analysts agreed on the label. The labeling accuracies were, in general, 
greater when there was agreement between the analysts. 

Table 3-2 shows a comparison, on a segment- by- segment basis, between accuracy 
obtained using the Reformatted Labeling Procedure and that obtained using the 
Integrated Labeling Procedure. Overall, the Reformatted Labeling Procedure 
produced labeling accuracies which were comparable to the accuracies for the 
Integrated Labeling Procedure. For some segments, the Reformatted Labeling 
Procedure obtained better results in certain categories than did the 
Integrated Labeling Procedure, while on other segments, the reverse was true. 
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TABLE 3-1.- ANALYST LABELING ACCURACY 
[Percent correctly labeled] 


Crop 

category 

All 

dots 

Pure 

dots 

Mixed 

dots 

Misregistered 

dots 

Nonsmal 1 
grains 

91(95) 

94(97) 

73(78) 

82(91) 

Small grains 
(except barley) 

m 

74(84) 


55(63) 

Barley 

B 

50(51) 

B 

50(-) 

Total small grains 


79(87) 


67(76) 


Note: The numbers In parentheses show the percentage of 
dots correctly labeled when both analysts agreed 
on the label. 
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TABLE 3-2.- SEGMENT-LEVEL RESULTS FOR SHAKEDOWN TEST 


Segment 

number 

Procedure 

Correctly labeled dots, % 

Segment A 
characteristics 


Bari ey 

Nonsmal 1 
grains 


Reformatted 

91 

•• 

93 

25% small grains 

1542 

TY 

42 

m 

96 

(no barley) 

3% other crops 


Reformatted 

86 

44 

88 

50% small grains 

1584 

TY 

93.4 

45 

94 

11% barley 

Acquisitions deficient 
for barley 


Reformatted 

57 

m 

95 

75% noncropland 

1656 

TY 

52.6 

- 

97 

7% small grains 
No barley 


Reformatted 

70 

31 

95 

38% small grains 

1664 

TY 

87 

54.5 

94 

8% barley 
27% other crops 


Reformatted 

56 

36 

81 

25% small grains 

1811 

TY 

70 

0 

94 

2% barley 
40% other crops 


Reformatted 

76 

52 

91 


Overall 

TY 

75 

55 

95 



Note: Segment 1514 was not processed during the TY. 
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The barley labeling accuracy was not very high for either procedure, with only 
half of the barley being labeled correctly. However, the segments involved in 
this test had an average barley proportion of only 5 percent, with two seg- 
ments containing no barley at all. Because of the nature of the barley /other- 
smal'^-grains labeMng technique, the labeling accuracy for barley cannot 
adequately tested if a reasonable amount of barley is not present. Therefore, 
in all of the subsequent discussions, barley is considered part of the small - 
grains category, and labeling accuracies are evaluated for small -grains/ 
nonsmall -grains labeling only. 

The labeling accuracies for Individual crops are shown in table 3-3. None of 
the nonsmall grain crops were consistently mislabeled, and of the small-grains 
crops, only flax was incorrectly labeled more often than it was correctly 
labeled. [This type jf labeling error for flax^ was observed in LACIE 
Phase III (ref. 8) and the Transition Year (ref. 9). Because there is so 
little flax, it is difficult on the basis of these and prior results to decide 
whether flax should be identified as a small grain or as a nonsmall grain.] 

3.3 CROPLAND/NONCROPLAND LABELING ACCURACY 

The labeling accuracy for the cropland/noncropland decision logic is shown in 
table 3-4. The dots considered in evaluating the cropland/noncropland label- 
ing accuracy were those which the analyst had decided were pure. The labeling 
accuracy obtained as a function of the path taken through the decision logic 
is also shown in table 3-4. None of the paths through the decision logic con- 
sistently produced wrong answers. It snould be noted that an affirmative 
response to question la occurred 84 percent of the time. In those instances 
when an affirmative was given, question 3 became the decision maker. While 
the labeling accuracy for the dots following this path which received a non- 
cropland label was consistent with the labeling accuracies for other pathways 
leading to a noncropland label, the labeling accuracy for the dots following 
this path which received a cropland label was lower than the labeling accuracy 



Although flax is not a small grain, its spectral signature is similar and is 
considered as grouped with the small grains. 
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TABLE 3-3.,- ANALYST LABELING ACCURACY FOR 

individual crops 


Crop 

Number of 
dots labeled 

Crops correctly 
labeled, % 
(nonsmall grains 
or small grains) 

Nonsmall grains; 



Alfalfa 

58 

81 

Corn 

155 

78 

Sunflower 

109 

92 

Sugar beets 

14 

79 

Grass 

112 

93 

Hay 


91 

Pasture 


95 

Trees 

12 

83 

Water 

34 

94 

Nonagricultural 

111 

96 

Homestead 

23 

87 

Idle 

257 

89 

Small Grains: 



Spring barley 

111 

83 

Spring wheat 

443 

81 

Flax 

34 

41 

Spring oats 

92 

62 

Ouram wheat 

16 

100 
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TABLE 3-4.- LABELING ACCURACY FOR CROPLAND /NONCROPLAND 
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Note: The numbers In parentheses reflect the percentage of labeled dots when both analysts 
agreed on the label. 







of the other pathway leading to a cropland label. Because 66 percent of the 
areas of these segments was cropland and because ti * labeling accuracy for the 
dots labeled cropland by decision 3 was lower than the labeling accuracy for 
the dots labeled noncropland. It can be seen that there were more noncropland 
dots Incorrectly labeled than there were cropland dots Incorrectly labeled. 
This, however, presents no later problem since the Incorrectly labeled 
noncropland dots remain In the flow of the decision logic. They may still be 
labeled nonsmall grains. In fact, for this reason one of these categories 
were to have a low labeling accuracy. It would be better that it be for the 
dots labeled cropland. Thus, the fact that the labeling accuracy for the dots 
labeled cropland Is lower than the labeling accuracy for dots labeled noncrop- 
land Is not disturbing. There did not appear to be any major problems with 
the cropland/noncropland decision logic. 

3.4 SMALL GRAINS/NOMSMALL GRAINS LABELING ACCURACY 

Table 3-5 shows the labeling accuracy for the small-gralns/nonsmall-gralns 
decision logic. The dots used In evaluating the small-gralns/nonsmall-gralns 
labeling accuracy are those which were correctly Identified as cropland by the 
analyst. The accuracy for this logic appears to be quite good, especially 
when there Is agreement between the analysts on the label. From the 
table 3-5(b), accuracy as a function of path through thj decision logic. It 
can be seen that a wide variety of paths through the logic are used. None of 
the paths appear to produce consistently Incorrect answers. This would 
Indicate that there are no major problems with the logic. 

As stated previously, there was not enough barley In these segments to deter- 
mine If the barley separation procedure was working properly. However, the 
accuracy In separating barley from other small grains Is presented In 
table 3-6. The dots used to determine this accuracy are those which were 
correctly labeled as small grains by the analyst. Only about half of the 
barley Is correctly labeled, while almost all of the other small grains are 
labeled correctly. 
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TABLE 3-5.- LABELING ACCURACY FOR SMALL GRAINS/MONSMALL GRAINS 


OF POOR QUALITY 


C3> 

O 


<a 

u 

a 

u 

u 

«s 


a» 

J3 


RS 

%. 

at 

> 

o 



0 
1/1 
u 

•S 

at 

5 

01 
a 
o 

i. 


4-> 

•a 


>> 

u 

<0 

k. 

a 

u 


>* 

































e 











at 











a 











c 

k 










at M 

at 











s 

4-> 



4-> 


4-1 




u 


«e 



IB 


IB 


4/1 


k 

4- 

at 



at 


at 


4.1 


1 /) at 

C 

f 


k 



x: 


IQ 


Q 

a 

» 


at 


X 


o 


E at 

«/i 



X 







u 


C71 


5 

ei 


o 


Ol 


x: a 


e 



c 


e 


c 


O TS 

c 



*4- 


e 



•F- 


f- o 

k 

k 

>t 

e 

k 

k 

k 


k 



o 

o. 

IQ 

a 

Q. 

o 

o. 


a. 


o 

m 

X 

CO 

to 

o 

CO 

1 

CO 

1 

V) 











a. 











o 











k 











o 











XI 











at 






















ot 






















<e »« 
















o 

0^ 


o 


o 

* 

VO 

00 

or 

in 

o 

o 

in 

o 

r»» 

o 

>» 

e> 


Ol 

Cl 

♦ H 

Cl 

Cl 


00 


•— 4-> 











•k> o 


PO 

r-» 



00 

rst 

in 

7-4 

in 

y "o 
at 

k 

Ol 


Cl 

Cl 

00 

00 

Cl 

eo 

00 

Cl 

k 











o 











o 











</) 











o 











at 

cst 

in 

00 

o 

r>. 

VO 

VO 

VO 

in 

CO 


CM 










le at 
*» ^ 
O (B 
h- ^ 











O) c 
c o 

</) 





C9 


cs 


ID 

C9 


C3 

CO 

o 

CO 

CO 

CO 

CO 

cs 

CO 

CO 

£ u 
fi at 
.j *e 

CO 

z 

CO 

in 

z 

CO 

z 

CO 

z 


VC 

>- 

• 

>- 

>- 

z 

>- 

Z 

1 

1 

1 

o in 

4-> 1/1 

' 

z 

1 

>- 

t 

>■ 

1 

>- 

1 

1 

c 











V/1 O ^ 
k 

at 

a ifl CO 

>- 

1 

1 

1 

1 

1 

>- 

1 

1 

1 

1 

1 

>- 

1 

1 

1 

1 

1 

M «t 

c a 

< O’ cu 

. 

>- 

>- 

1 

>- 

>- 

1 

>- 

1 

z 


>- 

>- 

>■ 

>- 

>- 

>- 

>- 

>• 

z 

>- 


3-9 


Note: The numbers in parentheses reflect the percentage of correctly labeled dots 

when both analysts agreed on the label. 

NSG = nonsmall grains 
SG = small grains 



TABLE 3-6.- LABELING ACCURACY 
FOR SMALL-GRAINS/BARLEY 
DISCRIMINATION 


crop type 

Small grains 

(except barley) 95(98) 

Barley 61(54) 
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3.5 CONSISTENCY OF ANALYST LABELING 


Ona of the important requirements for an objective labeling procedure is that 
it be consistent. In the Shakedown Test* each of the dots was labeled inde- 
pendently by two analysts. By comparing the results obtained by each of the 
analysts, the consistency of the procedure can be investigated. The first 
decision the analyst must make is whether the dot is pure or not. The results 
of this test showed that the analyst agreed on whether the dot was pure 
77 percent of the time. The analysts agreed on the final label for the dot 
85 percent of the time. Table 3-7 shows the consistency for the major steps 
in the procedure. Each of the percentage consistencies is based on those dots 
which were consistently labeled at the previous major step. The most inter- 
esting feature of these results is that the labeling is more consistent for 
pure dots (when the decision logic is used) than for mixed dots (which are 
labeled by comparison with pure dot labels). 

3.6 RECOMMENDATIONS FOR CHANGES TO THE REFORMATTED PROCEDURE 

The results of this test indicated that there were no major problems with the 
Reformatted Labeling Procedure. However, in order to determine if there could 
be some improvements to the procedure, an error characterization study was 
performed on the labeling from this test. The general conclusions from this 
study were that the consistent errors were due to atypical signatures and that 
there were no specific confusion crops. The error characterization did pro- 
vide some suggestions for changes which would improve the procedure and reduce 
the chances of clerical error. 

One of the most important recommendations concerned the handling of nonpure 
pixels. In the Reformatted Labeling Procedure, these pixels were reserved for 
labeling by imagery comparison after the pure pixels were labeled. This test 
showed that the labeling accuracy and consistency for these reserved pixels was 
less than for the pure pixels. Because of this difference, it was suggested 
that (if possible) the analyst should determine from the imagery which field to 
associate with the mixed pixel. Then a pure pixel should be designated in the 
field associated with the mixed pixel. The label for the mixed pixel could be 
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TABLE 3-7.- CONSISTENCY OF LABELING AS A FUNCTION OF 
DECISION LOGIC STEP 


Decision logic step 

Consistent labels, % 

Overall 

Pure dots 

Mixed dots 

Cropi and/noncropl and 

80 

85 

48 

Small grains/ nonsmall grains 

85 

95 

76 

Small grains/barley 

92 

94 

86 
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determined by applying the decision logic to the pure pixel associated with 
it. This should increase the labeling accuracy of the pure pixels associated 
with the mixed pixels to the same level ^s for the pure pixels. 

Another recommendation involved question 3 of the cropland/noncropland 
decision logic. This question determined the cropland/noncropland decision 
84 percent of the time and exhibited a lower labeling accuracy than did other 
paths. In addition, there was a certain amount of inconsistency in answering 
this question. The question asks whether the pixel is some shade of red on 
all acquisitions. It was recommended that the question be expressed in terms 
of the green number for the pixel rather than in terms of color on the 
imagery. This should make the question more objective. 

Recommendations were made for improving the clarity of the procedure and 
reducing clerical errors. In particular, the use of the time period A acqui- 
sitions in the small -grains/nonsmall -grains decision logic was not clear in 
the original procedure. Figure 3-2 shows the logic after it was revised to 
make use of the time period A acquisitions clearer. 

A number of review steps and internal consistency checks were incorporated 
into the label recording forms. This should help to eliminate clerical errors 
from the labeling process. 
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Figure 3-2,- Labeling logic for small gralns/nonsmall grains. 



4. CONCLUSION 


The results from the Shakedown Test indicated that there were no major prob- 
lems with the Reformatted Labeling Procedure as it was applied to the segments 
involved. The labeling accuracies were comparable with the accuracies for the 
Integrated Labeling Procedure. Though this performance needs to be verified 
through more extensive testing, the reformatted procedure does represent a 
substantial automation of the labeling process. With the recommended changes 
to the procedure, the Reformatted Labeling Procedure should be ready for 
testing on 1979 data. 
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DEVELOPMENT OF THE 
REFORMATTED SPRING SMALL GRAINS LABELING PROCEDURE 

Objective 

The objective of this effort was to develop a procedure for 
labeling small grains and barley in the northern U.S. Great 
Plains segments by converting the U.S. spring small grains and 
barley separation procedure used durirsg the Transition Project 
to a format similar to the corn/soybec.ns decision logic (Ref. ) 
The techniques that were used in the Transition Project were 
to be enhanced whenever possible. 

Approach 

Follov;ing a comprehensive reviev/ of the Transition Project 
labeling procedures, alternative methods for performing some of 
the steps were identified. These alternatives were designed to 
leave fev;er subjective analyst decisions in the labeling process 

The new techniques v;ere tested using segments from the 
developmental data set. Necessary modifications and revisions 
were made before incorporating them into the overall labeling 
procedure. 

Developmental Data Set 

The labeling procedure is based primarily on analysis/ 
observations of the segments shown in figure 1 v/hich comprised 


page 

quality 
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Figure 1,- Developmental data set 
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the developmental data set. Shaded areas on the map represent 
the major barley producing regions of each state. 

Criteria for selection of the segments were based upon having 
a sufficient number of acquisitions to adequately describe the 
growth cycle of spring small grains and having a reasonably 
large proportion of spring small grains, particularly barley. 

In South Dakota and Montana, an Intensive Test Site and two 
phase two bl:.nd sites v;ere used in order to obtain segments 
which were suitable for labeling procedure development. 

Discussion of the Procedure 

There are essentially three major divisions v/ithin the labeling 
procedure (appendix Al) . These are 1) the separation of dots into 
either cropland or non-cropland, 2) the separation of cropland 
dots into spring small grains or non-spring small grains, and 
3) the separation of spring small grains dots into barley or 
other spring small grains. 

For the cropland/non-cropland separation, the procedure relics 
on a slightly modified portion of the Decision Logic for Major 
Land-Use Categories v/hich v/as developed as part of the corn/ 

. I 

soybeans procedure. 
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TABLE 1.- EXPECTED CHARACTERISTICS OF ACQUISITIONS 
AS A FUNCTION OF WINDOW 


Window 

Description of Spring small grains 

Product 1 appearance of 
spring small grains 

1 

Plowing/planting for spring small 
grains 

All spring small grains appear to 
be bare soil 

Spring wheat Robertson stage 
0.8 - 2.4 

Light to dark green, 
light to dark gray, 
black 

2 

All spring small grains appear to 
be green vegetation. (Most of the 
summer crops appear to be bare 
soil. ) 

Spring wheat Robertson stage 
3.8 - 4.5 

Red, pink, brown 
r i ange 

3 

Spring barley is turned/harvested 
and spring wheat, oats, and flax 
appear to be green vegetation 

Deep red, reddish 
brown, brown, orange, 
pink, yellow, gold, 
olive, white, gray, 
green 

4 

Spring wheat Robertson stage 4.7 
to beginning of harvest 

All spring small grains appear to 
be turned/harvested. 

Light to dark green, 
light to dark gray, 
white, yellow, gold, 
olive, black 


PRECEDING PAGE BLANK NOT FILMED 
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window should allow accurate separation of spring snail grains 
from non-spring small grains. In an attempt to provide a more 
objective description oi: appearance, green numbers and bright- 
ness v/cre used in lieu of color descriptions for this procedure. 

Observation of the behavior of the green number/brightness of. 
spring small grains on segments from the developmental data set 
was used to establish the green numbe.v/brightness criteria for 
sp’-ing F..iZ\ll grains as a function of acquisition/v;indov;. These 
cutoffs were utilized in the decision logic for spring small 
grains. 

For the separation of barley and other spring small grains, 
much of the transition project labeling procedure was retained. 
However, there are several important modifications includij>g 
the follov;ing; 

1) The separation acquisition is selected using 
an objective procedure. This is the v;indow 3 
acquisition. 

2) The decision boundary on the green nimnber versus 
brightness scatter plot is a straight line V7ith 

fixed slope. 

' > 

3) The concept of dot drift is introduced to assist in 
determining the location of the decision boundary. 

Dot drift is the direction of movement in the green 
number-brightness plane from the v;indov/ 2 acquisi- 
tion to the window 3 acquisition. 

w pooa QUAUi^ 
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Minimum Acquiisition/vrinclow Reguire:nen“;.s 

The definition of a minimum data set for processing segments v;ith 
this labeling procedure reflects extensive LACIE experience in 
addition to observations of the segments from the developmental 
data set. 

A vd.ndov; 1 acquisition was knovm to be a requirement in mixed 
wheat areas to provide separation betv;een v;inter and spring 
small grains. This requirement v;as extended to all of the areas 
of interest because of its additional value for separating 
natural vegetation. 

An acquisition in window 2 or v/indow 3 is required to provide 
a date v;hen spring small grains are growing. Since the barley 
separation technique relies on observing barley turning/harvested 
while the other spring small grains are pre- turning, a v;indov; 3 
acquisition is required to execute that portion of the procedvire. 

An acquisition in window 4 is essential in areas such as South 
Dakota and Minnesota to avoid confusion between summer crops 
such as corn and spring small grains. 
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REFORIL\TTED SPRING S^JvLL GRraNS LABELING PROCEDURE 

The reformatted spring small grains labeling procedure is 
designed to be used for assigning labels to a pre-determined/ 
selected number of dots. Spectral data or statistics from 
these dot labels may be used as input to a machine classifica- 
tion/clustering algorithm. 

The general flow of the steps involved in the procedure is 
detailed in the diagram in figure 1. Following acquisition 
selection (step 1) , the combination of acquis itions/v.'indov;s 
available are considered to determine the type of labeling, 
if any, that can be performed using the procedure. 

If the available acquis itions/v;indows are sufficient for 
barley separation, the entire procedure can be executed. If 
an acquisition from windov; 3 is not available, only the spring 
small grains portion (steps 4 through 7) of the procedure can 
be used. 
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Select acquisitions 



Figure 1.- Flow Diagram of Reformatted Spring Small Grains Labeling procedure 
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1, Select AcTulsltlons 

Using the historical crop calendars for spring v;heat 
and spring barley, detcr.’nlne t>ie opening and closing 
dates for each of the follov,*ing four vrlndows: 


V/lndow 

Open 

Close 

1 

Spring V.'heat 

Flanted-5 days 

Spring V'hcp-t 50"? 
Planted i 18 days 

2 

Spring V.’hoat 
Headed -10 days 

Spring V/heat 50^C 
rlet'.ded + 10 days 

3 

Spring 3ar!ley 501? 
Turning to F:ipe - 
6 days 

Spring Barley 
Turning to Ripe + 
6 days 

4 

Spring V/heat 50?: 
Harvested + 15 days 

Spring '.vheat 50? 
Ha-rvested + 30 days 


Sort all available acquisition.-; covering the grovring 
season fpr spring sicall grains (beginning of plentlr.f 
to one Donth after the completion of harvest) into 
these wlndov.'s. 

Select one acquisition per v.’lndow. If more than one 
acquisition falls within a v:lndov:, select the one 
closest to the middle of the v;indovj. If two acqui- 
sitions are equidistant from the middle, select the 
latest cne. 

If a vrlndow does not contain an acquisition but one 
falls v?ithln three days of the opening or closing 
of the wlndov;, refer to thic adjusted crop calendar, 
meteoro] oglcal suirm:3.ries and location of the segment 
within the crop reporting district to determine 
whether or not the acquisition should be Included 
in the wlndov,-. 
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For exaniple, If an acquisition falls throe* days 
after the close of a window and the adjusted crop 
calendar/meteoroloeical sujiaarles indicate that in 
the a:*ea of the segment spring small grains are late 
developing or the segment is in the northernmost part 
of a large crop reporting district, Include the 
acquisition in the wlndov.*. 

In a similar manner, acquisitions falling within 
three days of the start or end of a v;indov? may 
be excluded from the wlndov:. Suppose an acquisition 
Is collected tv:o days before the close of window 1 
and the adjusted crop calendar/meteorologlcal 
summaries indicate that spring small grains 
develoT:^ent Is considerably ahead of normal In 
the area of the segment. The analyst should select 
another acquisition or if there are no other candidates, 
conclude that no windov: l acquisition exists. 

If available, the window 3 acquisition Is to be used 
as the base acquisition for labeling. If there is 
no v.'lndov: 3 acquisition, use the v.’indow 2 acqulsiticn. 

If neither of these windovjs contain an acquisition, 
the segment is unprocessable. 

Screen the base acquisition for data quality. If 
the acquisition contains excessive {>k0%) clouds, 
cloud shadows, haze or snow or other problems such 
as data dropouts, banding, etc,, revert to the second 
choice for the base acquisition. If data qu.alitj^ on 
the second choice is unacceptable, revert to the third 
choice. Continue until a base acquisition with 
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with acceptable data qxiallty h-.s been selected or 
the list of candidates has been exhausted. 

Screen each of the other selected acquisitions for 
data quality using the sanse criteria plus registration 
to the base acquisition to i one pixel. In each case, 
If the acquisition falls the data quality test, revert 
to the second choice, third choice, etc. until an 
accepr.able acquisition has been found or the candidates 
have been exhausted. 


The decision logic for spring small grains 

I 

requires the use of acqulsition( s) in addition 
to those previously selected if available. 

Acquisitions collected v.’ithln the time period beginning 
with the close of wlndovi 3 plus ^ 0 % of the distance 
betvreen the close of v.'lndow 3 and the opening of 
window 4 and ending vrith the opening of v.'lndov: 4 are 
described as being in time period A. This period 
is graphically described in Figure Z* 

The acquisitions selected and the time period A 
acquisitions should be recorded on the acquisition 
form as shovm in Figure 3 . The format of year, day 
should be 'used. 8124 indicates the 124^^ day of 19 ? 8 . 
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July I August I September \ 



Figure 2.- Graphical description of the 

of time period A. 
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Refer to the map in Figure 4 to deter:aine If the 
combination of v;indovrs/acquisi tions available meet 
the minimum requirements for processing. If the 
combination available is not listed as a processable 
data set, there is inadequate data for spring small 
grains labeling using this prc-cedure. 


3. Check for minimum data for bar] ev senar? tl on 

The barley separation procedure is based on the 
assumption that barley ripens and is harvested 
before spring vcheat, oats and flax. The acquisition 
selection process for selecting the windov; 3 
acquisition is intended to isolate the acquisition 
v:here this difference is maximised. Therefore, an 
acquisition in windov: 3 Is required for this 
procedure. 


A-17 



to 

■iJ 

a 

ty. 



c 


• O 

l.'l 

a 

u 

o 

i. 

c_ 


«c 

CJ 

i. 

3 

O) 

•r“ 

u. 


OR'GI!V/!l 

OF PG. i QUALITY 

4, Ca teporiZG ef< ch dot n s rmre , alyed « r.i s reri stored or 
obscurad~5v cXouciri. cloud sr.oacTs or i:u?.e . . 


The follovTlng definitions are used in this step: 


Pure dot - A dot v;hich is completely v.'lthin the 

same fleld/area on each of the selected 
acquisitions. 

Mixed dot - A dot v;hich is only partially within 
a fleld/area on the base acquisition, 

Misreglstered dot - A dot vjhlch is completely 

within a fleld/area on the base 
acquisition but shifts either 
partially or completely out of 
the fleld/area on one or core 
of the selected acquisitions. 


Using the base acquisition, locate the field/area 
associated with the dot of interest. If the pixel 
is not the same color as the field/area it is associated 
with, the dot should be considered mired. For example, 
in Figure 5f the dot of interest Is associated witli 
fiold A, a v;hlte field. If the pixel at this location 
appears pink rather than approximately the same color 
as the other pixels in field A, the dot should be 
considered mixed. 

If the dot is not mixed, the same test should be 
applied to the pixel at this location on each of the 
remaining selected acquisitions. If the dot shifts 
partially, or completely to another field, it should 
be considered misregistered. If the dot remains 
completely within the same field/area on all of the 
selected acquisitions., it should be considered pure. 
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Figure 5.- Relationship between a dot. and a field. 


A '- 20 


The determination of pure, mlyed or misreclstered 
should be recorded on the labellns form ns shovm 
in Figure 6. (P-pure, K-:nlxed, R-misreglstered) 

If a dot is found to be mixed, record the coordinates 
of an interior pixel from the field vrlth v;hich the 
dot is associated. 

If a dot is obscured by clouds, cloud shadovrs or 
haze on any of the selected acquisitions, leave 
the pure, mixed, misregistered column (column 13 ) 
blank and record a U in the final label column 
( column ) . 
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Figure 6.- Recording of pure, mixed, and misregister ed dots 
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5. Ser=ira‘ e nure riots cr op] -» r.ri or non -c ror>T -^nd 

"UsTi.'ir~t'**^ acqu: si frcrxecu a Troa wTTT:iC.w:;, 

execute the decision logic shc’vn in Flgu^'c 7 
each pure dot recording j'our responses in colutr.ns 
15 thru 19 of the labeling form as shov;n in Figure 
8 (Y-yes, N-no). 

If the decision logic indicates that the dot Is 
non-cropland, a D should be entered in the first 
label column (column ^5)* the dot is cropland, 

column 45 should be left blan'n at this point. 

The decision logic in Figure 7 is a portion of 
the Decision Logic for Major land-Uso Categories 
(Figure 9) which has been slightly modified for 
this procedure. The complete Decision Logic for 
Major Land-Use Categories can be fo\ind in Appendix 
B of the- Det;~ llnd Ana 3,vr;is Pre cf.riurc r f_o_2I '~^r --r!sl t i cn 
Pro ,1 e c t ( ?Y?9 ) . 
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Figure 7.- Decision Logic for Cropland - Non-cropland. 
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Figure 8.- Recording of responses from Cropland Decision Logic. 
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OF POOR Q'J;a!..1 s / 


6# SeT??i ivt v-Q Tpuro croolri?::^ riots into son ) nr soo 3 3 rrti i no 
end 1)0 “~rnrir^.r 11 rra ins . ”” 

For those pure dots determined to be cropland, 
execute the decision logrlc In Ficure 10. 

Those pure crop] and dots v*:hlch meet the £Tcen number/ 
brightness criteria for spring small grains on the 
acquisitions selected from wlnicv.'s arc subjected to 
a final test by requiring that the green number be 
less than 20 on all acquisltlcns collected during 
time period A. If the green number is not usable 
on these acquisitions due to misregistration, the 
dot should be reserved for lare] Ing along v.’ith the 
mli; 'd and mi sregi stored dots. 

The p.reen nurr.bors/brlghtncss values which arc used 
in majelng the decisions should V'O recorded in 
colur.vis 20 thru U-J of the lebelitjg form as shown 
in Figure 11. The labels of S for spring sn.^11 
grains and K for non-spring small grains should 
be recorded in columrj 45 , 
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Figure 10,- Decision Logic for pure cropland dots. 
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Figure 11.- Recording of pure cropland dots as spring small 
grains or non-spring small grains. 
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label , ■ jyc j , '''A 1 rtererl r rvcd Dots t'V 

coffioar • tc yu:-o o t s . 


Dellno-ate and annotate enou^^h of the field s/arc-as 
associated v;lth dots v;hich have been labeled D, K 
or S to provide a representative cross secticn of 
each class. Compare the Imarrery appearance (Product 1)* 
of each field/area associated with a mixed, mlsrecistere 
or reserved dot to the ennotated ficlds/arcnc and 
select the field/area which is most similar In 
appearance. Record the label of thr selected field/ 
area for the rairod, ml sreri stered or reserved dot. 

Record the labels in column 45 of the labeling form 
as shov.’n- in Figure 12. 


ORicijivj/iL r-'A(;dc fg 

OF POOR QUALITY 


* For instructions on the use of Product 3, refer to 
the De tfi 1 3 ed Analysis Procedures for Trr-nsitlon 
Pro.icct ; r fV9Y . ^ 
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Figure 12,- Recording of mixed, registered, and reserved dots as D, 












OF POOR QUALITY 


8» Labe], each rrirlrr; a:r;all rraln d ot as B » S,. j! or V, 

In column 46 of the labeling form, record one of the 

follov’ine labels for each spring small grain dot. 

(The recording Is illustrated in Figure 13*) 

B “ (barley) - spring small grains in the more 

advanced growth stages. (bright pink, yellov:, 
bright gold, tan, vfhlte, light gray, light 
green on Product 1 from v.indow 3)^^ 

S - (spring wheat, oats, flax) - spring small grains 
in the least advanced stages. (red, brown, 
reddish brov:n on Product 1 from windov; 

Q - spring small grains 'which appear to be between 
the groups labeDed 3 and S. Some spring wheat/ 
oats fields may be at the soft doug)j or ripe stagej 
as Illustrated in Figure 14, They v:lll not have 
a bright appearance but othierv.’i se may be coni' v.sed 
with barley. Dots vrhich fall into fields such 
as this should be labeled Q. 

V - spring small grains dots which were determined 
to be mixed unless they are associated with a 
field containing a dot labeled B or S. If they 
arej\they should receive the sane label as the 
pure dot. 

*• For instructions on the use of Product 3$ refer to 
the Detailed Analvsi s Procedures for Trc n s 1 1 1 on 
Fro.iect ( Fi?9 ) . 
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Figure 13.- Recorcmg of labels for scatter plot generation. 
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Figure 14.- General relationship between image appearance/growth stage and location 
on scatterplot generated from window 3 acquisition. 
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Conors n pto pi: r u V r; r 

of tho. L . 3 . ai-j d V dorr ii g i n ~ j. 


br r r. ■'; ,s a s ca *■. ^ o r.'”i o t 



Transfer the labels frou; coluir* k6 of the label ins ' 
form to a Process Request Porn and generate a 
green nuT.bcr versus brichtness scatter plot using 
the window 3 acquisition. If a window 2 acquisition 
is available, request green nv.niber versus brightness 
trajectory plots using the accuislticns froci 
wlndov/s 2 and 3 . (Additional acquisitions up to 
a total of eight nay be induced.) 

The relationship between t)iO location of a dot on 
the scatter plot and imagery cclor/grcvrth sta^e is 
generally as shov;n in Figure 34. The barley dots 
will fall to the right of the decision line and be 
widely scattered. The other spring sr.sll grains will 
form a relatively tight duster in the region noted 
as late headed to nilk doup.h. 
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10. I)s ter.iiine Do c 1 g 1 on 1. 1 no . 


OKKail\IAL S'A ..' ' 
OF POOR OUALl i v 


If thore ara B and S dots, cc si -uct a line on the 

scatter plot of the forni CII = 1.1 -r constant 

throufch the S dot v.'here the constant is a ctiniuuai 

and no pure B dots fall to t)ie left of the line (Line A). 

Construct a line of the forrii GX = 1.1 BH constant 

throurh tlie E dot whore th.c censtant is a carinurr. 

and no pure S dots fall to the right of the line (Lino B). 

(A template is provided to assist in constructlrjg 

these lines, ) 

If the location of the dots is such thot a line 
cannou be constructed, reex.-^n-ine the luage ayijyj&rance 
of the dot(s) which prevent cens truct j cn of tl'.o line. 

If the original label (s) were in error, charge rrie 
label(s) and continue. If the origin'.! ]abcl(r) are 
conflrned, place the line just to the right of the 
richtn.ost S dot in the case of Line 3 or Just to the 
left of the lcft:r.ost B cot in the case of Line A. 

If a v.’indcv: ?. acquisition is available, green nur-ber 
versus brightness trajectory plots 'will be used to 
assist in determining the decision line. Generally 
In the time period from Kindow 2 to vjlndov.' 3, barley 
dots beepme less green but brighter, The dot drift 
or direction of iiiovcment on the trajectory plot v;ill 
be do'wn and to the right. During this same perlc;d, 
spring v.'heat and oats dots become less green and 
less bright. The dot drift v;ill be down and to the loi t. 
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OF POOil Q-.Li'i ; 

If a vlndcw 2 acqulri ti ::n is e”k ilablo, transfer the 
dot drift frci the green nuiiher versus brightness 
trajectory plots to the scattc:' plot for each dot 
between lines A and B. Place the decision lint- 
parallel to and betv.’een lines A and B such that dots 
having different drift characteristics are separated. 
An erarcple of this is shovon in Figure 15 , 

If a wlridov; 2 acquisition is net avaiiable, place 
the decision line between and i-ara]]el to lines A 
and B such that 1 ) No dots to the right of the line 
appear to group with the S dots and 2) Dots to 
the right of the line are widely scattered as opposed 
to the closer knit group to the left of the line, 

This technique is lllustratcc? in Figure 16 , 

If no dots were labeled 3, construct L?ne A. If v-lndo 
2 acquisition is available, check the dot drift of 
dots v.’hlch fall to the right of the line to dctcriilne 
if they behave more like barley (increase in trightnes 
with decrease in grec-n number) or spring v:hoat(dccrcas 
In brightness v:lth decrease in green number). Use the 
dot drift, scatter as opposed to clustering and figure 
to detcra:ine if Line A should be the decision line 
or It should be to the right ,f and parallel to Lino A 
If no dots v;ere labeled S, construct Line B. Use the 
same technique described above to deternilne if Line B 
should be the decision line or whether it should be 
placed to the left of and paj-allel to J.ine B. 
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Figure 15.- Illustration of the use of dot drift characteristics 

to determine decision line. 
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Figure 16.- Illustration of the determination of 
decision line without a window 2 acquisition. 



11. PqI p bf’l Petr: !> r^r,ord tr-r to 3 or ^ t ^ on cf o r ^ i on 


P 4 


All Q and V dots which fall to the rid-ht of the doclsicn 
line should he lebe'jed B In thn final label coluTjn. 

All Q and V dots vrhich fall to the left of the decision 
line .should be labeled S in the final label colu-.r, . 

The original interpretation should be confirmed for 
any pure 3 dots v;hlch fall to the left of the 1 it;o 
and any pure S dots which fall to the rirht of the 
line. 

Final labels should be recorded on the label ing ferm 
in ccluicn as shov:n in Figure 17 , 
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Figure 17.- Recording of final labels. 
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PREFACE 


The Agriculture and Resources Inventory Survey*; Through Aerospace Remote Sensing 
is a multiyear program of research, development, evaluation, and application of 
aerospace remote sensing for agricultural resources, which began in fiscal year 
1980, This program is a cooperative effort of the U.S. Department of 
Agriculture, the National Aeronautics and Space Administration, the National 
Oceanic and Atmospheric Administration (U.S. i .. tment of Commerce), the Agency 
for International Development (U.S. Department of State), and the 
U.S. Department of the Interior. 

The work which is the subject of this document was performed by the Earth 
Resources Applications Division, Space and Life Sciences Directorate, Lyndon B. 
Johnson Space Center, National Aeronautics and Space Administration and Lockheed 
Engineering and Management Services Company, Inc. The tasks performed by 
Lockheed Engineering and Management Services Company, Inc., were accomplished 
under Contract NAS 9-15800. 
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1. BACKGROUND 


The Foreign Commodity Production Forecasting project of the Agriculture and 
Resources Inventory Surveys Through Aerospace Remote Sensing (AgRISTARS) pro- 
gram was responsible ^’or developing and testing procedures for using aerospace 
remote sensing technology to provide more objective, timely, and reliable crop 
production forecasts. One of the component? of production estimation is 
segment area estimation. Since large-area acreage estimates for small grains 
depend upon segment-level proportion estimates, it is important that those 
proportion estimates be as accurate and precise as possible. Prior to the 
AgRISTARS program, several procedures were tested in an attempt to find an 
accurate and efficient method for estimating small-grain proportions. In the 
resultant method. Procedure 1 (PI), labels were used in the random selection of 
training pixels to start a clustering algorit.m. Then, cluster statistics were 
used to produce a maximum likelihood classification of the scene into 2- or 
3-class strata. Finally, stratified proportion estimates were made using a 
second random set of labeled dots. However, this classification component 
provided no better results than those which could have been produced through 
simple random sampling. Thus, clustering had not been an effective method. 

Consequently, a new clustering algorithm was developed (refs. 1 and 2). 
Previously, clusters were used to define distributions in the data. The new 
algorithm used clusters to generate strata within which crop proportions could 
be estimated. One advantage of this algorithm was that, as an unsupervised 
routine, a first set of trair'ng dots was not needed (as in PI). 

In addition, a proportion estimation technique (ref. 3) which used the dusters 
of this algorithm was developed. This technique involved Bayesian estimation 
of cluster-level proportions based on historical information concerning cluster 
purities. The cluster-level estimates were then weighted by their relative 
cluster sizes and aggregated to produce the segment-level estimate. Use of 
this technique was expected to provide better proportion estimates. The tech- 
nique also implemented sequential sampling in an attempt to sample the segment 
clusters more effectively and further reduce the expected mean squared error 
(MSE) of the proportion estimation. 
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Characteristic of this new estimation technique, the Bayesian Sequential 
Allocation/Bayesian Estimator (BSA/BE), was the selection of dots, one at a 
time. The sampling technique was an attempt to minimize the MSE of the propor- 
tion estimate. Before each sampling of a dot, expected effects to MSE estimates 
were made for each cluster; and, on the basis of these estimates, a sample was 
taken from the cluster that was expected to most reduce the MSE. This manner of 
sampling provided an additional feature; the option of sampling with a fixed 
sample size or varying the sample size from segment to segment. Varying the 
sample size could be managed by halting the sampling when a predetermined 
threshold was obtained for the internal MSE estimate. Varying sample sizes in 
this manner was to provide uniform accuracy across segments by sampling more 
frequently from more "difficult" segments. 

A 10-segment development test of the BSA/BE (ref. 4) showed that there was at 
least a 2-to-l reduction in the MSE from that observed from PI, a reduction in 
proportion estimatioti error, and improved analyst labeling accuracy. 
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2. APPROACH 


Flow diagrams of the BSA/BE technique and PI are presented in figures 2-1 and 
2-2, respectively. Table 2-1 shows the four steps involved in stratified areal 
estimation ,rid a comparison of the BSA/BE to PI at each level. The BSA/BE dif- 
fers from el at three of the four steps; whereas PI makes use of approximately 
proportional allocation of sample dots to Iterative Self-Organizing Clustering 
System (ISOCLS) clusters and a relative count estimator of cluster-level propor- 
tions, the BSA/BE technique makes use of sequential allocation of sample dots to 
CLASSY clusters and a Bayesian estimator of cluster-level proportions. By 
incorporating only step 1 of the BSA/BE into PI (that is, by substituting CLASSY 
clustering for ISOCLS clustering) and proportionally allocating sample dots to 
clusters based on cluster sizes, a new estimation technique, the Proportional 
Allocation/Relative Count Estimator (PA/RCE) is defined. By additionally incor- 
porating step 3 of the BSA/BE, the Proportional Allocation/Bayesian Estimator 
(PA/BE) technique is defined. Both of these techniques were included for test- 
ing in this experiment. A fourth technique, the Random S«mpl i ng/Rel ati ve Count 
Estimator (RS/RCE), was also included in the experiment. The RS/RCE, which ran- 
domly samples the entire scene without regard to clusters and employs a relative 
count estimator of segment-level proportions, was included since PI had not 
proved to be significantly better than the RS/RCE. The PA/RCE was included to 
determine the effectiveness of CLASSY clustering. The PA/BE was included to 
determine the effect of the cluster-level Bayesian estimator with proportional 
allocation. 

For each of these four techniques, the dot sets that were input had labels from 
one of three possible sources: the integrated labeling procedure (ref. 5), the 

reformatted labeling procedure (ref. 6), or ground-truth data. Combining the 
four techniques with the three sources of dot labels and the two sample size 
requirements (fixed or variable), 24 estimates were made for each segment. Tr^ 
effect of these three factors on the estimates was to be determined. 
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Figure 2-1.- Segment analysis using Bayes Sequential Allocation procedure. 
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Figure 2-2.- Segment analysis, Procedu 







Examination of the effects of the different techniques will, in essence, measure 
(a) the effect of using stratified random sampling of CLASSY clusters, which are 
proportional to cluster size, in estimating spring small-grain proportions 
rather than randomly sampling the entire scene; (b) the effect of Bayesian 
procedures rather than relative frequency in estimating proportions at the 
cluster level proportions; and (c) the effect of Bayesian Sequential Allocation 
rather than proportional allocation in estimating spring small-grain proportions 
(ref. 7). 
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3. METHOD 


The dot sets from which samples were taken contained dots on one of the four 
major grids or alternates for grid dots. Enough dots were labeled from each 
segment so that 75 dots were allocated proportional ly to the clusters; this was 
usually the 209 dots from the first grid plus a few (1 to 10) from grid 2. 

This was to insure that each cluster would have enough dots for sequential 
allocations. If it was determined that a grid dot was a boundary dot, an 
alternate dot was substituted for labeling purposes since boundary dots present 
special labeling problems; pure dots have been found to have higher labeling 
accuracies than do boundary dots, but to ignore them by using only pure grid 
dots in proportion estimation could bias results (refs. 8 and 9). From these 
dot sets, sample dots were taken for proportion estimation. 

Two separate estimation processings were made for 35 spring wheat segments: 
for one, a fixed sample size of 50 dots was used; and for the other, varying 
sampling sizes from segment to segment were allowed. 

To permit variable sample sizes, two dots were automatically allocated to each 

cluster so that MSE estimates could be obtained. Then, a threshold was set on 

^ 2 

the internal segment MSE estimate (MSE = E(p - p) < .0020). When this thres- 
hold was reached, sampling was halted. To achieve comparable results using 
other techniques, this same sample size was applied to them to obtain propor- 
tion estimates. Thus, while the sample size could vary from segment to seg- 
ment, it was constant among the techniques by which estimates were made for any 
particular segment. 
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4. RESULTS 


Because there were insufficient data (only nine segments were processible using 
the reformatted procedure) on which to base an evaluation when the reformatted 
labeling procedure was used, the part of the evaluation which would include 
that procedure will not be considered. In appendix A, however, the results are 
presented for the four estimation techniques for which labels were obtained 
from the reformatted procedure. Only those results which were obtained when 
the integrated procedure labels or ground-truth labels were input were 
considered in the evaluation. 

Although estimates were made with fixed and variable sample sizes, emphasis 
during the evaluation was placed on the fixed sample case. Results of the 
variable sample case were comparable to those of the fixed sample case; these 
results, which include biases, MSE's, and plots of proportion estimation 
errors, are presented in appendix B. Further discussion of the analysis and 
results will concern only the fixed sample case for input dot sets with labels 
from the integrated procedure or ground-truth data. 

Tables 4-1 and 4-2 present biases of proportion estimates, standard deviations 
of estimate errors, and MSE's for all 35 segments when dot labels from the 
integrated procedure were input. The errors are shown in figure 4-1 
(ground-truth proportions for these segments are presented in appendix C). 

On the basis of analyst-interpreter (AI) labels, the PA/RCE technique provided 
a significantly less biased estimate and produced less variable errors than did 
random sampling. The fact that the errors were less variable showed that the 
clustering algorithm had been effective. 

When ground-truth labels were input, the errors prodLced using the PA/RCE were 
less variable than those of random sampling (table 4-1 and figure 4-2); but, 
the disturbing res ’lt was the significant bias produced by random sampling. 

With ground-truth labels input, random sampling was expected to provide an 
unbiased estimate. Ground-truth labels were input to determine the effect of 
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TABLE 4-1.- ACCURACY ANU PRECISION OF THE INTEGRATED 
PROCEDURE WITH AI LABELS AND GROUND-TRUTH LABELS 


Technique 

AI labels 

Ground-truth 
label s 

Bias 

Standard 

deviation 

MSE 

Bias 

Standard 

deviation 

MSE 

Random Sampling/ 

Relative Count Estimator 

-5.7 

7.7 

90 

-2.5 

6.9 

53 

Proportional/ 

Relative Count Estimator 

-4.0 

6.2 

53 

0.0 

4.0 

16 

Proportional Allocation/ 
Bayesian Estimator 

-3.5 

6.0 

47 

0.5 

3.8 

14 

Bayesian Sequential Allocation/ 
Bayesian Estimator 

-2.7 

6.8 

52 

0.4 

4.7 

22 


4-2 


















ORIGIMAL PAii'c 5S 
OF POOR QUALITY 

TABLE 4-2.- RELATIVE ACCURACY AND PRECISION OF THE INTEGRATED 
PROCEDURE WITH AT LABELS AND GROUND-TRUTH LABELS 



AI labels 


Ground-truth 
label s 

Technique 

7 . 

P 

(a) 

— 
Relative 
bias, % 

(b) 

RV 

(c) 

7 

P 

(a) 

Relative 
bias, % 

(b) 

RV 

(c) 

Random Sampling/ 

Relative Count Estimator 

23.4 

-24.4 

32.9 

26.6 

9.4 

25.9 

Proportional / 

Relative Count Estimator 

25.1 

-15.9 

24.7 

29.1 

0.0 

13.7 

Proportional Allocation/ 
Bayesian Estimator 

25. £ 

-13.7 

23.4 

29.6 

1,7 

12.8 

Bayesian Sequential Allocation/ 
Bayesian Estimator 

26.4 

-10.2 

25.8 

29.5 

1.4 

15.9 


^Average proportion estimate = p 

^Relative bias = x 100% 

P 

A 

Op 

CRV = 100 X -^ = relative variation 

•X 

P 
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Ground-truth proportion 














techniquef with unbiased estimators on the variability of error's and the effect 
of techniques with biased estimators on both the proportion estimates and the 
variability of errors. However, random sampling as an unbiased technique, pro- 
duced a significant underestimate even when ground-truth labels were Input. To 
determine the reason for this result, the biases of the 209-plus pixel 1n!^ut 
dot sets were examined since these were the sets from which the 50-dot samples 
were taken. The bias (over all 35 segments) was found to be -0.8 percent, and 
the estimate produced by random sampling was not really significantly biased 
with respect to this. This Indicates that the use of the PA/RCE technique 
resulted In the overestimation of the 209-plus dot proportion estimates by 
0.8 percent. While this was not a significant overestimate. It should be 
noted. The Important result achieved was the reduction of error variability 
produced by the PA/RCE from random sampling when A1 labels and ground-truth 
labels were input. This reduction was attributed to CLASSY clustering. 

Cluster purities are further discussed in appendix 0. 

Since clustering was effective, the next step was to determine the effect of a 
Bayesian estimator. For the PA/BE, the same dots that were used for the PA/RCE 
were again used. Thus, the only difference between the two techniques was the 
estimator employed; with the PA/BE, a cluster-level Bayesian estimator was used 
Instead of a relative count estimator. It had been hypothesized that the PA/BE 
would provide Improved proportion estimates over the PA/RCE because prior know- 
ledge of cluster purities was being considered. Such results could be expected 
In the same way that the PA/RCE was expected to provide proportion estimates 
that were more accurate than those obtained through random sampling because of 
the use of clustering Information. As hypothesized, there seemed to be 
improved precision; but, the difference was small (table 4-1). Figure 4-3 
shows the difference between the PA/BE and the PA/RCE for all 35 segments. 

A positive difference indicates that the PA/BE produced the larger' estimate. 

As the PA/RCE estimate Increased, there was a tendency for a larger positive 
difference. Whether AI labels or ground-truth labels were input, the PA/BE 
produced a mean proportion estimate that was five-tenths of a percent larger 
than that of the PA/RCE. This was attributed to a tendency for positive 
biasing (with respect to the PA/RCE) by the Bayesian estimator (figure 4-3). 
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The net effect was a reduction of a negative bias when AI labels were input. 
With the positive biasing, however, the result was a slight reduction (0.2 
percent) in error variability from that of the PA/RCE. This was the case when 
AI labels were input and also when ground-truth labels were input. In both 
cases, the MSE's of the PA/BE were slightly reduced from those of the PA/RCE. 
These results were encouraging because they supported the expectation that 
Bayesian estimation at the cluster ‘.evel would provide greater precision 
(although producing slightly biased results) over maximum likelihood 
estimation. 

The final technique was the BSA/BE, the results for which (as can be seen in 
table 4-1) showed it to be the least biased technique when AI labels were 
input. This had been hypothesized since the dots were allocated to clusters 
one at a time with the intention of minimizing the MSE, Although it produced 
the least biased results as hypothesized, the BSA/BE produced more variable 
results than did proportional allocation. This was a disturbing observation. 

In an effort to further study these results, an attempt was made to separate 
the effects of Bayesian estimation and sequential allocation. In order to 
determine whether or not the results of the BSA/BE followed those of the PA/BE 
when compared to an unbiased estimation technique, estimates were made using 
the r^ame sequentially allocated dots and cluster information with a relative 
count cluster-level (BSA/RCE) estimator rather than the Bayesian estimator. 
Using the Bayesian estimator in the proportion estimation process increased the 
estimates by approximately 2 percent. This was true whether input labels were 
from AI's or ground-truth data (table 4-3). As in proportional allocation, 
Bayesian estimation produced less variable results at the expense of biasing. 
However, with sequential allocation, this bias was not as slight as with pro- 
portional allocation. A graph comparing the two sequential estimates for each 
of the 35 segments is presented in figure 4-4. Notice that there was greater 
overestimation for segments with lesser amounts of small grain. 
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TABLE 4-3.- ACCURACY AND PRECISION OF SEQUENTIAL ALLOCATION 


Technique 

AI labels 

Ground- truth 
1 abel s 

Bias 

standard 

deviation 

MSE 

Bias 

Standard 

deviation 

MSE 

Sequential allocation 
(relative count, 
cluster-level estimate) 

1 

7.1 

73 

-1.7 

5.3 

30 

Sequential allocation 
(Bayesian cluster-level 
estimate) 


6.8 

52 

+0.4 

4.7 

22 
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The fact that the BSA/BE produced more variable results than did the PA/BE was 
due, in part, to a decreased overall labeling accuracy (table 4*4). In order 
to determine whether or not these differences were significant, the differences 
between labeling accuracies of the samples for each segment from those of all 
labeled dots for each segment were found. The means of these differences are 
shown in table 4-5, While there was a significant improvement of small-grain 
labeling accuracy, there was a simultaneous decrease in nonsmall -grain labeling 
accuracy. The result was a slight decline in total labeling accuracy. 

These results indicate that, with a small sample of 50 dots, proportional allo- 
cation is the sampling method that produces the most precise and reliable esti- 
mates. A slight reduction in variability can be gained at the cost of slight 
biasing of results by using the Bayesian estimation technique. 

Although CLASSY clustering was effective (that is, proportional allocation of 
dots to CLASSY clusters resulted in greater precision for a given sample size), 
the same precision could be obtained by random sampling without the need of 
clustering information if a large enough sample size were taken. If dot sets 
with AI labels were input with the present labeling accuracy, a random sample 
of 85 dots would be required to obtain the precision of 50 dots proportionally 
sampled from CLASSY clusters. If labeling was perfect, a random sample of 
166 dots would be required to obtain the same precision of 50 dots 
proportional ly allocated to CLASSY clusters. 

Therefore, the biases of proportion estimates, standard deviations of errors, 
and MSE's of all available labeled dots from the 209 pixels were found when dot 
sets with AI labels were input and when dot sets with ground-truth labels were 
input. Table 4-6 presents the results obtained when those dots were treated as 
a random sample. It was expected that these dots would provide greater preci- 
sion than a 50-dot proportional sampling of CLASSY clusters because of the 
larger sample size. Just as we expected, when using all available labeled 
dots, the RS/RCE showed less variable errors than the PA/RCE when it used only 
50-dot samples allocated to CLASSY clusters. Notice in table 4-6 that the use 
of alternate dots did not introduce a bias; the mean error was very small when 
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ground'truth labels were used. This was Important since analysts substituted 
alternate dots for boundary dots In both the Integrated and reformatted label- 
ing procedures to provide better labeling targets to eliminate the special 
labeling problems that boundary dots present. 

In order to determine the effect of clustering with larger samples, cluster- 
level proportion estimates were made with a relative count estimator on the 
basis of all labeled dots and weighted by their cluster sizes to produce seg- 
ment-level estimates. These results are shown In table 4-7. As can be seen, 
clustering had little effect on the accuracy or precision of estimates when 
these larger samples were taken. These results point to labeling errors as the 
limiting element In precision. 
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TABLE 4-7.- ACCURACY AND PRECISION OF ALL LABELED 
DOTS WHEN WEIGHTED BY CLUSTER SIZE 


Dots 

AI labels 

Ground truth 
labels 

Bias 

standard 

deviation 

MSE 

Bias 

standard 

deviation 

MSE 

All labeled dots (wefghted) 

BB 

5.7 

48 

BW 

2.5 

6.3 

All labeled dots (random) 

-3.9 

5.8 

48 

-0.8 

2.9 

9 

Proportional sampling 

BB 

6.2 

53 


4.0 

16 
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5. SUMMARY AND CONCLUSIONS 


For the first time in Foreign Commodity Production Forecasting (FCPF) project 
testing, clustering has been an effective method in making proportion 
estimates. Proportionally allocating 50 dots to CLASSY clusters to estimate 
proportions resulted in greater precision than using a random sampling of 
50 dots. This was observed when dot sets with AI labels from the integrated 
procedure were input, and it was also observed when dot sets with ground-truth 
labels were input. 

When a cluster-level Bayesian estimator (rather than a relative count estimator) 
was employed with proportional allocation, errors of proportion estimates were 
slightly less variable at the expense of a slight positive bias with respect to 
the estimate of the PA/RCE technique. When dot sets with AI labels from the 
integrated procedure were input, the results of the PA/BE were less biased with 
respect to ground-truth proportions. Whether analyst-! abeled dot sets or 
ground-truth labeled dot sets were Input, the net result was a reduction in the 
MSE. 

The BSA/BE provided the least amount of bias with respect to ground-truth pro- 
portions when analyst-labeled dot sets were input. However, this was due to 
positive biasing by the Bayesian estimator with respect to an unbiased estimate 
based on the same dots, also weighted by cluster size. The magnitude of this 
bias was approximately 2 percent. This same effect was observed when dot sets 
with ground-truth labels were input. In addition, the errors of estimates from 
the Sequential Bayesian technique showed greater variability than did those 
from proportional sampling. This was attributed, in part, to a reduced overall 
labeling accuracy observed for dots selected through sequential allocation. 
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It was estimated that In order to obtain the same precision with random sampl- 
ing as obtained by the proportional sampling of 50 dots with an unbiased esti- 
mator, samples of 85 or 166 would need to be taken If dots sets with AI labels 
(Integrated procedure) or ground-truth labels, respectively, were Input. 

Little difference, on the other hand, was observed between random sampling and 
duster-weighted estimates when all available labeled dot from the 209 were 
Input. Another Important result Is that dot relocation by analysts provided 
dot sets that were unbiased. 


6. RECOMMENDATIONS 


While automatic labeling would provide large samples at relatively low costs. 

It Is only a goal. With large samples, these clustering procedures do not seem 
to provide much Improvement In proportion estimation. However, It Is not 
recommended that effective clustering algorithms be discarded. Neither should 
efforts In proportion estimation techniques be defaulted to random sampling. 

An effective procedure using clustering Information Is available for use In 
testing and for future development. Automatic labeling. It should be remem- 
bered, Is not yet a reality. It Is therefore recommended that these proportion 
estimation techniques be maintained, particularly the PA/BE because It provided 
the greatest precision. It Is recommended also that this estimation procedure 
be considered as the base line for the 1981-82 FCPF Spring Small Grains Pilot 
Experiment. Further exploratory testing needs to be conducted for other crops 
of Interest such as corn and soybeans. 
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APPENDIX A 

RESULTS OF THE FOUR ESTIMATION TECHNIQUES UNDER REFORMATTED PROCEDURE 

Because of biowindow restrictions, only nine segments were processible under 
the reformatted procedure. Biases of proportion estimates (for fixed samples) 
along with standard deviations and mean-squared-errors (MSE's) for these seg- 
ments are presented in table A-1. The errors of the proportion estimates are 
shown in figures A-1 and A-2. When dot sets with labels from the reformatted 
procedure were input, large positive biases were produced through the use of 
all the techniques. Although the estimates produced by techniques using CLASSY 
clustering were less biased, there was no significant difference among the 
biases because of the great amount of variation in the errors; as can be seen, 
the standard deviation of the proiortion estimate errors in each of the tech- 
niques was approximately 19 percent. Errors in the labeling of dots and the 
limited number of segments would not permit enough of a basis to warrant an 
evaluation of the techniques when labels result from the Reformatted procedure. 
But to be complete, comparable statistics are provided in table A-1 for these 
same segments when ground-truth labels, were used. Interestingly, the standard 
deviations and MSE's were smaller when CLASSY clustering was used. 

TABLE A-1.- ACCURACY AND PRECISION OF THE REFORMAHED 
PROCEDURE WITH AI LABELS AND GROUND- TRUTH LABELS 


Technique 

AI labels 

Ground-truth 

labels 

Bias 

Standard 

deviation 

MSE 

Bias 

standard 

deviation 

MSE 

Random sampling/ 

Relative Count Estimator 

9.1 

19.4 

436 

-0.8 

6.1 

36 

Proportional Allocation/ 
Relative Count Estimator 

6.2 

19.2 

382 

-1.5 

3.9 

17 

Proportional Allocation/ 
Bayesian Estimator 

6.0 

18.8 

369 

-1.7 

3.9 

17 

Bayesian Sequential Allocation/ 
Bayesian Estimator 

6.3 

19.1 

381 


4.0 

22 
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Figure A-2.- Proportion estimation results with ground-truth labels. 
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APPENDIX B 


RESULTS OF FOUR ESTIMATION TECHNIQUES UNDER VARIABLE SAMPLING OF SEGMENTS 

Proportion estimates for segments with varying sample sizes were made only when 
dot labels were obtained from the Integrated procedure or ground-truth data. 

In table B-1, biases, standard deviations, and MSE's for proportion estimates 
made under sampling br.sed on a threshold (set at .0020) for an Internal MSE 
estimate are presented. 

Proportion errors are shown In figures B-1 and B-2. The results were similar 
to those of the fixed sample size. The sample sizes averaged approximately 
42 dots and ranged from 25 to 75 dots. 
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TABLE B-1.- BIASES AND VARIANCES 


Ground-truth 

labels 

Internal 
NSE estimatp 

42 

43 

17 

20 

1 NSE 

50 

33 

32 

24 

Standard 

deviation 

00 00 P- 

• • • • 

VO ir> LT> ^ 

Bias 

-2.3 

-0.4 

+0.1 

+0.3 


1 Internal 
[msE estimate 

42 

45 

18 

20 

o> 

(0 

MSE 

83 

48 

43 

53 

< 

Standard 

deviation 

6*9 

6*5 

1*9 

1*8 


Bias 

-4.4 

-3.5 

+2.9 

-2.5 

Technique 

Random sampling/ 

Relative Count Estimator 

Proportional Allocation/ 
Relative Count Estimator 

Proportional Allocation/ 
Bayesian Estimator 

Bayesian Sequential Allocation/ 
Bayesian Estimator 
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Figure B-1.- Proportion estimation results with analyst labels 
under varying sample sizes. 





Random Sample/ Proportional Allocation/ 

Relative Count Estimator Relative Count Estimator 


ORIGINAL PAGE IS 
OF POOR QUALITY 



Figure B-2.- Proportion estimation results with ground-truth labels 

under varying sample sizes. 
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APPENDIX C 

1979 GROUND-TRUTH PROPORTIONS 


Segment 

Ground-truth 

type 

(a) 

Barley, % 

Other spring 
small grains, % 
(b) 

Total spring 
small grains, % 

1387 

D 

8.01 

35.36 

43.37 

1392 

D 

2.02 

28.28 

30.30 

1394 

I 

0.31 

39.51 

39.82 

1457 

I 

3.15 

38.24 

41.39 

1461 

I 

4.99 

48.19 

53.18 

1467 

0 

3.09 

48.46 

51.55 

1472 

I 

4.02 

35.16 

39.18 

1473 

D 

11.69 

39.74 

51.43 

1485 

I 

1.35 

20.80 

22.15 

1514 

D 

4.92 

22.77 

27.69 

1518 

D 

0.29 

25.22 

25.51 

1524 

D 

0.00 

6.96 

6.96 

1571 

I 

0.32 

14.60 

14.92 

1612 

I 

0.00 

16.03 

16.03 

1617 

D 

21.18 

39.68 

60.86 

1619 

0 

10.39 

39.76 

50.15 

1627 

I 

0.00 

15.80 

15.80 

1630 

I 

0.67 

16.80 

17.47 

1636 

I 

0.87 

38.91 

39.87 

1653 

I 

0.00 

16.13 

16.13 

1658 

I 

1.44 

32.41 

33.85 

1664 

D 

1.94 

33.50 

35.44 

1676 

I 

0.23 

7.44 

7.67 

1755 

I 

6.55 

5.64 

12.19 

1784 

I 

4.07 

17.29 

21.36 

1825 

0 

6.20 

19.95 

26.15 

1835 

0 

5.61 

19.02 

24.63 

1843 

D 

0.75 

5.13 

5.88 

1909 

I 

0.88 

17.15 

18.03 

1918 

I 

1.14 

13.80 

14.94 

1920 

I 

0.09 

21.11 

21.20 

1924 

I 

1.01 

36.75 

37.76 

1948 

D 

1.95 

5.57 

7.52 

1974 

I 

4.48 

35.25 

39.73 

1987 

D 

15.48 

34.40 

49.88 


®D Indicates 400 dot ground-truth proportions. 

I indicates inventoried ground-truth proportions from universal 
. ground-truth tapes. 

^Other spring small grains include spring wheat, oats, durum wheat, 
and flax. 
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APPENDIX D 
CLUSTER PURITIES 

In order to determine the appropriateness of a beta prior for cluster propor- 
tion estimates, small -grain proportions for each cluster were found from 
ground-truth data* The percentage of all clusters having small-grain propor- 
tions within f1 ve-hundreth Intervals was then foun 1. These clusters are shown 
In figure D-1. The continuous line represents th« shape of a beta prior with a 
mean equal to the mean small -grain proportion estimate for tho.je segments 
(0.26). Thus the beta prior Is given as follcws: 

where a ■ 0.3513 and 3*1. 

As can be seen, the beta seems to be a reasonable prior. 
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PREFACE 


The Agriculture and Resources Inventory Surveys Through Aerospace Remote 
Sensing Is a 6-year program of research, development, evaluation, and appli- 
cation of aerospace remote sensing for agricultural resources which began In 
fiscal year I960. This program Is a cooperative effort of the National Aero- 
nautics and Space Administration, the U.S. Agency for International Develop- 
ment, and the U.S. Departments of Agriculture, Commerce, and the Interior. 

The research which Is the subject of this document was performed within the 
Earth Resources Applications, Space and Life Sciences Directorate, at the 
Lyndon B. Johnson Space Center, National Aeronautics and Space Adminis- 
tration. Under Contract NAS 9-15800, personnel of Lockheed Engineering and 
Management Services Company, Inc., performed the tasks which contributed to the 
completion of this research. 

The following Individuals contributed to this effort: Dr. A. G. Houston, NASA, 
helped with his Interest and suggestions. M. L. Sestak, Lockheed, put together 
the original data set, and Dr. P. Doralswainy, Lockheed, was responsible for the 
model Improvements and much of the Information on the Inner workings of the 
models. 
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1. BACKGROUND AND OBJECTIVES 


This report describes the results of the evaluation of the performances of can- 
didate agrometeorological crop calendar models. These models have been pro- 
posed by the Supporting Research Crop Calendar Project element for possible 
application to labeling procedures of the Agriculture and Resources Inventory 
through Aerospace Remote Sensing (AgRISTARS) program. This study Is an 
addition to the 1980 U.S. and Canada Spring Wheat and Barley Exploratory 
Experiment. 

During the Large Area Crop Inventory Experiment (LACIE), spring wheat planting 
date and crop development stage estimates based on historical normals were 
Improved by the use of the Feyerherm planting date and Robertson Spring Wheat 
Crop Calendar Models. Modifications were subsequently made to the Robertson 
model to Improve deficiencies identified In LACIE evaluations. These 
modifications were tested along with a state-of-the-art barley model (Williams, 
ref. 1) which became available for testing for the first time. 

This study Investigated two crop planting date (or starter) models, namely the 
Feyerherm (ref. 2) and the Normal models (ref. 3), and four crop growth stage 
models. These crop development stage models are designated the Original 
Robertson Model (RO), the Improved Robertson Version 1 Model (Rl), the Improved 
Robertson Version 2 Model (R2), and the Williams Barley Model. The evaluation 
was based on 1979 ground- truth data consisting of 49 spring small grains blind- 
site segments In the U.S. Great Plains region and contains three crop 
categories of Interest, spring wheat, durum wheat, and barley. For the 
purposes of this study, durum wheat Is In the same category as spring wheat. 

The primary objective of this study was to determi ne the combination of the 
crop planting date model and the crop development stage model which would most 
accurately predict the crop development stage as a function of time for spring 
wheat and barley. Other objectives were to determine If the Williams model 
predicts more accurately the barley development stages than do the Robertson 
models and to determine whether the models selected would produce results which 
are sufficiently accurate to be used In labeling and classification procedures. 
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2. APPROACH 


The Feyerherm and the Normal planting date models were evaluated on their 
ability to accurately predict the median planting dates for the segments. The 
basis for comparison was the ground- truth median planting dates which yielded 
errors measured In units of days associated with the models. The ground-truth 
median planting dates for the spring wheat crop and for the barley crop were 
obtained by calculating the date at which SO percent of the spring wheat or the 
barley fields In each of the segments were observed to be planted. 

The performances of the three Robertson development stage models were evaluated 
using the ground- truth median development stages as the basis for comparison by 
use of the observed median planting dates to Initiate the models. The ground- 
truth median development stages for the spring wheat crop and for the barley 
crop were obtained by calculating the observed median stage for the spring 
wheat or the barley fields within each of the segments for each of the dates on 
which the stages were observed. The comparison of the models' predictions 
versus the observed crop stage yielded errors In terms of crop stages 
associated with each of the models. 

The barley development stage model was evaluated using the observed median 
planting dates for barley to Initiate the models and subsequently to compare 
the model prediction of stage with the ground- truth median development stages 
for barley. 

The planting date models and the development stage models were evaluated 
Independently so as to minimize any adverse consequences to the performances of 
the crop development stage models as a result of Inaccurate planting date Input 
to the models. 

Certain assumptions had to be made regarding the ground- truth data used for 
evaluation. The 49 segments contained from 15 to 30 special fields that were 
distributed through the segment and observed periodically. The locations and 
selections of these special fields were assumed to be random, and the 
periodically observed stages were assumed to be truly representative of crop 
development at those times. 
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3. DATA SET 


The data set used In this study comprised 49 blind-site segments In the spring 
wheat areas of the U.S. Great Plains and 1979 periodic observations collected 
by enumerators at 9- to 18-day Intervals corresponding to Landsat overpass 
dates (ref. 4). These periodic observations contained planting dates and crop 
development stages for each field In the segment. The number of fields within 
a segment varied from 15 to 30 spring wheat or barley fields. The planting 
date model contained the observed planting dates and predicted planting oates 
for spring wheat and barley. The crop stage model data contained observed crop 
stages and predicted crop stages for each of the models. The crop stages were 
given in terms of the Robertson Phonological Crop Scale. 

Figure 1 Is an Illustration of the Robertson Phonological Crop Scale that was 
used in this study, superimposed on the Feekes Scale description of Identifi- 
able crop stages (refs. 5 and 6). Figure 2 shows the geographic location of 
the segments that contain the data set used In. this study. 


3-1 


RIKNING 

STEM EXTENSION — HEADING — ^ HARVEST 


OF POOR QUALITY 



Figure 1.- Illustration of the Robertson Phenological Scale. 
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Figure 2^- Geographic locations of the segments. 


4. DESCRIPTION OF THE MODELS 


Robertson's concept (ref. 5) Is based on certain physiological processes that 
are central to the development of spring wheat. Since temperature and photo- 
period are two primary environmental factors that Influence the phenological 
development, a photothermal concept was used to compute the development of a 
crop over five fairly short and uniform physiological periods. The trlquadra- 
tlc responses of temperature and photo-period were estimated for each of the 
phenological stages by an Interative regression technique. 

The Improved Robertson Version 1 and the Improved Robertson Version 2 Models 
are Improvements over the Original Robertson Model with respect to the 
photo-period and temperature responses. The photo-period response Is limited 
to stages between emergence and flowering. The thermal responses for sub- 
sequent stages are adjusted to represent realistic physiological responses. 

The development rates of spring wheat Immediately before and after flowering 
are responsive primarily to the dally maximum temperature. 

The Williams Barley Model Is based on approximately the same concept as the 
Robertson model, the difference being that the coefficients were developed 
specifically for barley. 

Figure 3 Is a schematic of the models' Input requirements and resulting output 
data. The Normal model, although not an agrometeorological model. Is Included 
In figure 3 for the sake of completeness. It Is based on historical data 
averaged for the crop reporting district. The dally minimum and maximum 
temperatures are obtained from reports of weather stations nearest the segments. 
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Figure 3.- Schematic of the evaluation 




5. RESULTS FOR PLANTING DATE MODELS 


Both the Feyerherm and the Normal models produce median planting date estimates 
at the segment level. The performances of the models for the spring wheat 
fields and the barley fields were evaluated separately. 

Figure 4 Is a histogram showing the distribution of errors measured In days for 
the Feyerherm versus the Normal planting date models applied to spring wheat 
fields. The error Is the difference between the median ground- truth planting 
dates and ;he model -predicted planting dates, and the distribution of these 
errors should give an Indication of the bias associated with the models. As 
can be seen from figure 4, both distributions appear normal, the differences 
being the locations of the midpoints of these distributions. The Feyerherm 
model has a positive displacement, whereas the Normal model has a negative dis- 
placement. This Indicates that the Normal model Is very early compared to the 
ground- truth median planting dates, while the Feyerherm model Is slightly 
late. Based on reports jointly published by the U.S. Departme:-t of Agriculture 
and the National Oceanic and Atmospheric Administration In the Weekly Weather 
and Crop Bulletin, the 50 percent planting date of spring wheat in North Dakota 
for 1979 was 13 days late. Thu:., the Normal mooel performed as expected. 

Table 1 summarizes the statistics on the errors measured in days for the 
Feyerherm versus the Normal model applied to spring wheat. The sign test shown 
1n table 1 Is based on the absolute magnitude of the error and gives the 
percent of times one model Is closer to the ground-truth than the other model. 

TABLE 1.- COMPARISON OF ERRORS IN PLANTING DATE MODELS 
APPLIED TO SPRING WHEAT FIELDS 



Feyerherm model 

Norma! model 

Number of segments (n) 

49 

49 

Mean error (In days) 

+3.9 

-10.4 

Standard deviation (In days) 

7.0 

7.50 

Median error (in days) 

+4.0 

-11.0 

Sign test {%) 

75.5 

22.4 

(2% tied) 
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From table 1 It can be seen that, on the average, the Feyerherm model Is 
3.9 days late compared to the observed planting date, whereas the Normal is on 
the average 10.4 days early compared to the observed planting date. Statistics- 
ally, the sign test Indicates that the Feyerherm model Is significantly better 
than the Normal model at the 6-percent level of significance. The overall 
statistics Indicate that the Feyerherm model Is closer to the ground-truth than 
the Normal model In predicting spring wheat planting dates for this year. 

Figure 5 Is a histogram showing the distribution of error measured In days for 
the Feyerherm versus the Normal planting date models applied to barley fields. 
As can be seen from figure 5, both distributions appear normal. However, the 
Feyerherm model midpoint has a positive displacement, whereas the Normal model 
has a negative displacement. This Indicates that the two models are, on the 
average, late and early compared to the ground- truth median planting dates av 
seen for barley fields. 

Table 2 summarizes the statistics on the error measured In days from the 
Feyerherm versus the Normal model applied to barley fields. From table 2, It 
can be seen that, on the average, the Feyerherm model Is 2.9 days later than 
the observed planting date, whereas the Normal Is, on the average, 10.9 days 
earlier than the ob'^erved planting date. The sign test Indicates that the 
Feyerherm model Is better than the Normal model, though not statistically sig- 
nificant at the 5-percent level of significance. The overall statistics 
Indicate that the Feyerherm model Is better for this year than the Normal model 
Is for predicting barley planting dates. 

TABLE 2.- COMPARISON OF ERRORS IN PLANTING DATE MODELS 
APPLIED TO BARLEY FIELDS 



Feyerherm model 

Normal model 

Number of segments (n) 

44.0 

44.0 

Mean error (in days) 

+2.9 

-10.9 

Standard Deviation (In days) 

11.48 

9.55 

Median error (in days) 

+4.5 

-11.5 

Sign test 

63.6 

36.4 
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6. APPROACH: CROP DEVELOPMENT STAGE MODELS 


The three Robertson models and the Williams Barley Model were started using the 
ground- truth median planting dates for spring wheat and barley fields as Input 
to the models. They were evaluated on their ability to accurately predict 
median crop development stages for spring wheat and barley between stages 2.0 
and 6.0, which are the emergence through ripe stages. 


In an attempt to determine if the models performed differently during different 
parts of the growing season, the models were evaluated at five ranges of stages 
as shown below. 


1. Stage 2.0 to 2.9 

2. Stage 3.0 to 3.9 

3. Stage 4.0 to 4.9 

4. Stage 5.0 to 5.9 

5. Stage 6.0: 


emergence to prejointing 
jointing to preheading 
heading to presoft dough 
soft-dough to preripening 
ripe 


In addition, the overall performance was tested for the entire growing season 
from stage 2.0 to stage 6.0. 
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7. CROP DEVELOPMENT STAGE MODEL RESULTS APPLIED TO SPRING WHEAT FIELDS 


Figure 6 contains scatter plots of the median predicted development stages 
versus the observed median development stages for models RO, Rl, and R2. The 
letters represent the number of data points falling on the character 
(A » 1, B ■ 2, etc.). The common trend on all three plots Is for the predicted 
growth stage to converge on the 1-1 line, Indicating that the performance of 
all three models Is Improving with time through the growing season. It can 
also be seen from figure 6 that model RO Is progressing faster than models R1 
and R2 by noting that 13 ground truth observations are off scale and greater 
than stage 6.0 (I.e., swathed and harvested). 

Table 3 summarizes the statistics on the errors between the observed stages and 
the predicted stages that were applied to spring wheat at various Intervals 
throughout the growing season. The errors are the differences between the 
predicted stages and the observed stages and should give an Indication of the 
amount of bias associated with each of the models. An average positive error 
would Indicate that the model Is ahead of the ground-truth , while an average 
negative error would Indicate that the model was behind the ground- truth. In 
addition, the absolute value of the error was ranked on a scale of 1 to k, 
where k Is the number of models being compared with each other (In table 3, 
k = 3). The sum of the various ranks associated with each model was then uti- 
lized In a Friedman nonparametric test of ranks (ref. 7) to determine If any 
one model produced better results consistently. 

Table 3 shows that there were no significant differences between any of the 
three models when evaluating the overall performance from ground-truth stages 
2.0 to 6.0. The range of the mean error for the three models was two-tenths of 
a stage, and the Friedman T-statistIc also Indicates that there is no 
significant difference between the models at the 95-percent confidence level. 

For stages 2.0 to 2.9, there was a marginal difference between the three 
models. It is apparent that R1 Is the worst performer of the three models at 
this stage Interval, as Indicated by the statistics on the errors and the 
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TABLE 3.- COMPARISON OF ROBERTSON MODELS APPLIED TO SPRING WHEAT 


Ground-truth 

range 

Statistic 

Robertson 0 

Robertson 1 

Robertson 2 

2.0 - 6.0 

Mean error 

0.0 

0.2 

0.2 

Enti re 

STD 

0.53 

0.48 

0.46 

growing 

Median error 

0.0 

0.1 

0.2 

season 

tRank observed 

100.21 

97.08 

96.71 


Friedman's T-statistic: 0.15 (not significant) 

2.0 - 2.9 

Mean error 

0.9 

1.0 

0.9 


STD 

0.25 

0.28 

0.25 


Median error 

0.9 

1.0 

0.9 


ERank observed 

25.00 

37.75 

27.25 


Friedman's T-statistic: 6.17 (significant) 

3.0 - 3.9 

Mean error 

0.3 

0.7 

0.4 


STD 

0.26 

0.32 

0.26 


Median error 

0.3 

0.7 

0.4 


ERank observed 

42.42 

95.25 

66,33 


Friedman's T-statistic: 41.17 (significant) 

4.0 - 4.9 

Mean error 

-0.2 

0.1 

0.1 


STD 

0.26 

0.27 

0.31 


Median error 

-0.2 

0.1 

0.0 


ERank observed 

89.67 

70.75 

79.58 


Friedman's T-statistic: 4.48 (not significant) 

5.0 - 5.9 

Mean error 

-9.2 

0.0 

0.1 


STD 

0.42 

0.27 

0.33 


Median error 

-0.2 

0.0 

0.2 


ERank observed 

109.45 

66. GO 

93.95 


Friedman's T-statistic: 20.92 (significant) 

6.0 

Mean error 


M •» 



STD 

« mm 


m m 


Median error 

-0.5 

-0.4 

-0.3 


ERank observed 

50.0 

48.4 

33.5 



Friedman's T-statistic: 24,07 (significant) 


At 95-percent confidence level, 
At 99-percent confidence level, 


Friedman's T-statistic critical value 
Friedman's T-statistic critical value 


5.99. 

9.21. 
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Figure 6.- Scatter plots of predicted versus ground-observed spring wheat 
crop stages for the three spring wheat crop stage models. 


observed sum of the ranks. From stages 3.0 to 3.9, there was a significant 
difference between the models. RO appeared to be the best at this stage 
Interval. From stages 4.0 to 4.9, there was no significant difference between 
the models. 

For stages 5.0 to 5.9, there was a significant difference between the models, 
and R1 appeared to perform the best within this stage Interval. Finally, at 
stage 6.0, there was a significant difference between the three models, and R2 
appeared to perform the best of the three models. At ground-truth stage 6.0, 
the mean and standard deviation have not been displayed, as they are not valid. 
The observations obtained beyond stage 6.0 were beyond the range of the model's 
abilities of prediction and, therefore, were not valid. 


8. CROP DEVELOPMENT STAGE MODEL RESULTS APPLIED TO BARLEY FIELDS 


Figure 7 contains scatter plots of tne median predicted development stage for 
model R2 and the Williams Barley Model versus the observed median development 
stage. The letters represent the number of data points falling on that 
character. At first glance, there Is no apparent difference between the two 
models, although the barley model appears to be more dispersed about the I-l 
line than R2 (figure 10). More significant Is the fact that 33 observations 
are beyond 6.0, Indicating that the barley model Is progressing faster than the 
spring wheat model. 

Table 4 gives the statistics on the errors between the median ground-truth 
stage and the model predicted median stage applied to barley at various stage 
Intervals through the growing season. It can be seen from table 4 that there 
was a significant difference between the models for the overall performances 
from stages 2.0 to 6.0. The barley model Is significantly worse than at least 
one of the spring wheat models. 

From stage 2.0 to 2.9, there were marginal differences between the models. RO 
appeared to perform the best of the four models as Indicated by the error sta- 
tistics and the observed sum of the ranks. For stages 3.0 to 3.9, there was a 
significant difference between the models. RO appeared to be the best of the 
four models. From stages 4.0 to 4.9, there were no significant differences 
between the models. They appeared to be nearly Identical at this stage Inter- 
val. For stages 5.0 to 5.9, there was a significant difference between the 
models. Model R1 appe^ 'ed to perform the best. At stage 6.0, there were no 
significant differences between the models, and R2 appeared to perform the best. 
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TABLE 4.- COMPARISON OF ROBERTSON AND WILLIAMS MODELS APPLIED TO BARLEY 


Ground-truth 

range 

Statistic 

Robertson 0 

Robertson 1 

Robertson 2 

Willi ams 
barl ey 

o 

• 

AO 

1 

o 

• 

CM 

Mean error 


0.0 

0.0 

0.4 

Enti re 

STD 


0.60 

0.61 

0.60 

growi ng 

Median error 


0.0 

0.0 

0.0 

season 

ERank observed 

mSBEm 

96.96 

98.58 

126.79 


Friedman's T-statistic: 8.74 (significant) 

2.0 - 2.9 

Mean error 

1.0 

1.1 


1.2 


STD 

0.32 

0.37 


0.35 


Median error 

1.1 

1.2 


1.2 


ERank observed 

22.33 

33.50 

24.67 

39.50 


Friedman's T-si 

tatistic: 9.4 

^9 (significant) 

3.0 - 3.9 

Mean error 

0.3 

0.4 

0.4 

0.6 


STD 

0.32 

0.38 

0.36 

0.42 


Median error 

0.2 

0.4 

0.3 

0.5 


ERank observed 

50.58 

90.67 

65.08 

113.67 


Friedman's T-s‘ 

tatistic: 43.79 (significant) 

4.0 - 4.9 

Mean error 

-0.3 

-0.1 

■BPS 

0.1 


STD 

0.32 

0.34 


0.52 


Median error 

-0.3 

0.0 

-0.1 

0.2 


ERank observed 

89.42 

62.67 

74.92 

79.0 


Friedman's T-statistic: 7.] 

L8 (not significant) 

5.0 - 5.9 

Mean error 

-0.5 


-0.2 



STD 

0.57 

0.45 

0.54 



Median error 

-0.6 

-0.2 

-0.2 



ERank observed 

129.93 

70.67 

95.10 

MSSEm 


Friedman's T-statistic: 28.68 (significant) 

6.0 

Mean error 



— 

MM 


STD 

-- 

— 

— 



Median error 

-0.9 

-0.7 

-0.6 



ERank observed 

48.0 

35.0 

26.5 



Friedman's T-statistic: 14.31 (significant) 


At 95-percent confidence level, Friedman's T-statistic critical value = 7.82. 
At 99-percent confidence level, Friedman's T-statistic critical value = 11.34. 
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Figure ?.- Scatter plots of predicted versus ground-observed barley crop stages for the 
Williams Barley Model and the Improved Robertson Version 2 Spring Wheat Model. 


9. APPLICATION TO LABELING PROCEDURES 

The results shown 1n the preceding discussion Indicate that the Feyerherm 
planting date model Is more accurate than the Normal model. However, with 
respect to the growth-stage models. It Is not readily apparent that any one 
model produces consistently better results through the growing season. The 
Improved Robertson Version 2 Model was selected on the basis of Its being the 
most physiologically realistic model for application to the labeling procedures 
In AgRISTARS (ref. 8). In order that the models be useful for the spring small 
gra'ins labeling procedure. It Is necessary that they be able to predict crop 
growth stages at particular points of time with reasonable accuracy. The 
Reformatted procedure (ref. 9) prescribes and Identifies four Landsat acquisi- 
tion biowindows that are necessary for accurate labeling as shown In table 5. 

TABLE 5.- BIOWINDOWS FOR REFORMATTEL PROCEDURE 


W1 ndow 

Open 

Close 

1 

Spring wheat 50?. 
Planted minus 5 days 

Spring wheat 50% 
Planted plus 18 days 

2 

Spring wheat 50% 
Headed minus 10 days 

Spring wheat 50% 
Headed plus 10 days 

3 

Spring barley 50% 

Turning to ripe minus 5 days 

Spring barley 50% 

Turning to ripe plus 6 days 

4 

Spring wheat 50% 
Harvested plus 15 days 

Spring wheat 50% 
Harvested plus 30 days 


Using the criteria described In table 5, the predicted growth stages for the 
Improved Robertson Version 2 Model were converted to days of development to 
reach each of three crop stages described In the Reformatted procedure. 
Biowindow 4 was not calculated because It was out of the ranges of stages In 
which the models are effective. 


Table 6 lists the median ground- truth dates and the median predicted dates for 
three biowindows. 31ow1ndow 1 used the Feyerherm planting date model and bio- 
windows 2. and 3 used the Improved Robertson Version 2 model for spring wheat 
and barley with the Feyerherm planting date model as the starter model. 
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TABLE 6.- OBSERVED VERSUS PREDICTED BIOWINDOWS 
ACCORDING TO THE REFORMATTED PROCEDURE 


OBS 

STATE 

APb 

CRD 

County 

Seg. no. 

Blowlndow 

OBSPLT 

1 (SH) 
FPLT 

Bi.iWlndow 2 (SH) 
08 SHE AO R2HEAD 


n 

27 

15 

70 

Redwood 

1380 

126 

136 

183 

190 

e 

« 


27 

19 

40 

Grant 

1566 

136 

140 

186 

191 

232 

2.36 

H 

27 

19 

40 

otter Tall 

183b 

139 

145 

194 

196 

221 

240 


27 

19 

40 

Yellow Medicine 

1842 

123 

134 

191 

164 

• 

a 


27 

20 

10 

Marshal 1 

1514 

158 

148 

211 

198 

236 

2JP 


27 

20 

10 

Roseau 

1518 

148 

148 

201 

197 

246 

?if, 

7 

27 

20 

10 

Norman 

1825 

142 

144 

196 

197 

219 

146 

6 

27 

20 

10 

Polk 

1987 

133 

145 

191 

196 

214 

23d 

9 

30 

23 

SO 

Fergus 

1948 

136 

123 

188 

188 

719 

220 

10 

30 

104 

10 

FI athead 

1725 

120 

115 

192 

175 

223 

223 

11 

38 

19 

20 

Benson 

1392 

153 

148 

198 

197 

226 

239 

12 

38 

19 

20 

Pierce 

1461 

147 

155 

204 

202 

226 

249 

13 

38 

19 

20 

Bottineau 

1611 

155 

155 

204 

202 

223 

241 

14 

38 

19 

20 

McHenry 

1612 

146 

150 

195 

196 



IS 

38 

19 

30 

Ramsey 


152 

153 

202 

202 

220 

242 

16 

38 

19 

30 

Towner 


155 

159 

197 

7.09 

223 

257 

17 

38 

19 

30 

Cavalier 

1617 

155 

154 

214 

202 

247 

242 

19 

38 

19 

so 

Stutsman 

1636 

143 

144 

202 

193 

229 

237 

19 

38 

?9 

60 

Barnes 

1472 

145 

148 

196 

200 

212 

241 

20 

38 

19 

90 

01 ckey 

1658 

133 

142 

193 

195 

217 

233 

21 

38 

19 

90 

Sargent 

1664 

141 

145 

191 

194 

207 

236 

22 

38 

19 

90 

La Moure 

1924 

143 

144 

196 

194 

226 

238 

2’ 

38 

20 

30 

Pembina 

1584 

159 

147 

213 

197 

226 

240 

24 

38 

20 

30 

Grand Fork 

1619 

135 

146 

201 

196 

219 

238 

2S 

38 

20 

60 

Cass 

1473 

141 

142 

200 

192 

229 

237 

26 

38 

20 

60 

Traill 

1645 

143 

142 

196 

192 

228 

237 

27 

38 

20 

90 

Richland 

1399 

136 

144 

183 

196 

205 


28 

38 

20 

90 

Ransom 

1974 

140 

145 

198 

197 

220 


29 

38 

21 

10 

Burke 

1394 

154 

156 

201 

199 

225 

241 

30 

3b 

21 

10 

Hard 

1457 

156 

159 

202 

209 

232 

246 

31 

38 

21 

10 

Mountrail 

1602 

152 

158 

204 

207 

247 

246 

32 

38 

21 

40 

Dunn 

■ is 

136 

145 

187 

193 

232 

241 

33 

38 

21 

40 

McKenzie 

■ Ifl 

141 

138 

187 

185 



34 

38 

21 

40 

Mercer 


149 

145 

198 

168 

240 

226 

35 

38 

21 

50 

Kidder 

1909 

140 

148 

198 

198 

214 

240 

36 

38 

21 

70 

Hettinger 

1650 

136 

141 

186 

190 



37 

38 

21 

80 

Burleigh 

1653 

142 

152 

197 

201 

• 


38 

38 

21 

80 

Morton 

1656 

.1.43 

149 

195 

199 

204 

242 

39 

38 

21 

80 

Eninons 

1917 

138 

136 

186 

188 

222 

226 

40 

38 

21 

80 

Grant 

1918 

131 

149 

191 

199 

217 


41 

38 

21 

80 

Sioux 

1920 

134 

136 

186 

188 

. 


42 

38 

21 

90 

McIntosh 

1661 

137 

147 

193 

199 

220 

238 

43 

46 

15 

60 

Ml nnehalia 

1784 

123 

134 

173 

189 

210 

228 

44 

46 

16 

SO 

Brule 

1676 

118 

121 

183 

e 

201 

219 

45 

46 

16 

50 

Sully 

1689 

no 

127 

180 

184 

220 

222 

46 

46 

16 

SO 

Jerauld 

1755 

109 

128 

175 

183 

198 

223 

47 

46 

17 

10 

Oewey 

1485 

123 

134 

185 

187 

212 

230 

48 

46 

19 

20 

Ednainds 

1599 

140 

136 

184 

189 

214 

226 

49 

46 

19 

30 

Roberts 

1960 

132 

140 

188 

189 

209 

236 


OBSPLT 

■ Observed planting date 

CiBSRIPL ■ Observed ripening date 


(ground-truth). 

(ground-truth) . 

FPLT 

• Feywherm planting d»*e 

R2RIPE ■ Improved Robertson 


(preuSwkeo). 

Version 2 Model 
ripening date (predicted) 

08SHEAO 

■ Observed heading date 
(predlcteo). 


R2HEAO 

■ Improved Robertson Virslon 
2 Model heading date 
(predicted). 
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Figure 8 shows how the models would perform in biowindow 1 {planting stage), 

2 (heading stage), and 3 (ripening stage for bar.ey) If the criteria described 
In table 5 for the reformatted procedure were applied to the Feyerherm and 
Improved Robertson Version 2 Models. For example, In figure 8 the vertical 
lines are the limits of the biowindow prescribed In the procedure. The verti- 
cal distance between these two lines Is the width of the window In days for the 
biowindow (In this case, the window Is 23 days). Each horizontal bar repre- 
sents the location of the biowindow predicted by the model for each of the 
49 segments. 

In figure 8 for biowindow 1, It can be seen that there is a fair amount of 
overlap with the prescribed biowindow with a bias towards the positive side 
(I.e., the model Is progressing faster than the observed stage). For 
biowindow 2, there Is a fair amount of overlap with little apparent bias. For 
biowindow 3, there Is poor overlap with a bias on the positive side. Table 7 
gives the probability that the model prediction will be within the prescribed 
biowindow. This was achieved by dividing the total number of days predicted 
Inside the ground- truth window by the total number of days within the window 
for all the segments. It can be seen from table 7 that the Feyerherm model is 
accurate In predicting the planting data for biowindow 1 (spring wheat plant- 
ing) 73 percent of the time, the Improved Robertson Version 2 Model selects 
days in biowindow 2 (spring wheat heading) 73 percent of the time and In bio- 
window 3 (barley ripening) only 21 percent of the time. 

TABLE 7.- REFORMATTED PROCEDURE BIOWINDOW SELECTION RESULTS 



Biowindow 1 
(spring wheat, 
plant) 

Biowindow 2 
(spring wheat, 
head) 

Biowindow 3 
(barley, ripe) 

Total percent outside window 

27.0 

27.0 

79.0 

Percent days past the window 
(model late) 

22.0 

15.0 

75.0 

Percent days before the window 
(model early) 

5.0 

12.0 

4.0 

Probability of being Inside window 

73.0 

73.0 

21.0 
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Figure 8.- Comparison of predicted biowindows versus the ground-observed 
biowindows for the Reformatted procedure. 



10. DISCUSSION OF RESULTS AND CONCLUSIONS 


It should be noted that the analysis described Is based on only 1 year of 
ground- truth data. It Is possible that the scale utilized for the ground- truth 
data may be too coarse it half a stage) to be used In this type of analysis. 
This Is evident from tables 3 and 4 where the errors are. on the average, one- 
tenth to seven- tenths of a stage off. A small shift In the ground truth could 
conceivably shift the results to yield a different set of conclusions. 

As far as the Feyerherm and Normal planting date models are concerned, the 
Feyerherm model Is closer to the true planting date, as can be seen from the 
results. It Is the more realistic of the two models because It compensates for 
unusual spring planting conditions whereas the Normal model does not. The 1979 
crop year was unusual In that both spring wheat and barley were planted later 
than usual (ref. 7). 

There appeared to be no difference between any of the spring wheat models 
(I.e., RO, Rl, and R2) applied to spring wheat, based on the ground- truth data 
available for evaluation. The differences in the magnitudes of the errors 
between the three models are so small as to be Insignificant from a physical 
standpoint, as can be seen from tables 3 and 4. This Is true for almost all 
the stage Intervals within which the models were evaluated. Since the ground- 
truth data are no more accurate than a one-half stage, any differences in the 
models could probably be attributed to "noise.'* The same may be said of the 
Robertson and Williams models when they are applied to barley so far as the 
magnitudes of the errors are concerned. It can be seen from figure 7 that the 
Williams model Is progressing too fast for barley. 

So far as application to the Reformatted procedure is concerned, the Feyerherm 
model performs adequately for the planting stage while the Improved Robertson 
Version 2 Model performs adequately for the heading stage but not for the 
ripening stage. 
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Modifications to the Original Robertson Model yielded more accurate results at 
the later stages of spring wheat growth than the earlier stages. An example of 
the improvement in performance can be seen in figure 9 which shows the distri- 
bution of the errors for stages 5.0 to 5.9. Both the improved versions show a 
smaller amount of variability than the Original Robertson Model. 
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Figure 9.- Distribution of crop stage prediction errors for spring wheat stages 5,0 - 5.9. 



11. RECOMMENDATIONS 


Based on the results to date, it 1s recommended that the Feyerherm planting 
date mode! be utilized for both spring wheat and barley. It appears that the 
Improved Robertson Version 2 Model is the more useful for predicting spring 
wheat and barley development stages. However, the model is not adequate to 
determine window 3 of the Reformatted procedure, which is used to separate 
barley from spring wheat. Further research on biowindow 3 is required if 
accurate results are to be obtained for identifying this window. 
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